MagerValp

AutoDMG v1.9

2019-09-24T00:00:00+00:00

AutoDMG v1.9 is out! Apple unfortunately introduced a couple of OS installer bugs in Mojave beta 4 that are still unresolved. I’ve implemented a workaround for one of them, but one still remains:

Added 10.14 support.
⚠️ To successfully build on Mojave the installer must be on an external volume, see issue #222.
Removed 10.10 and 10.11 support.
Removed HFS+ support for 10.14+.

You’ll find the download on the AutoDMG releases page. If you have questions or need help, you can join us in #autodmg on Slack MacAdmins (but please check the documentation and especially the FAQ first).

AutoDMG v1.8.1

2018-04-24T00:00:00+00:00

I’ve released AutoDMG 1.8.1, which includes the following bugfixes:

Add App Transport Security exception for cdn-apple.com.
Show a helpful error message when an incomplete installer is used.
Better error messages when image building fails.
Fail but don’t crash if a profile is bad.

defaults write with double slash

2018-01-30T00:00:00+00:00

My ass has been bitten twice by this now, so I figured a blog post to warn others might be in order. Say you’re writing a package postinstall script where you want to set a preference, and you dutifully use $3 to target the installation volume. You might write a script along these lines:

#!/bin/bash
# This is buggy, don't use it. A working script is at the bottom of this post.

defaults write "$3/Library/Preferences/com.tyrell.nexussix" lifespan -int 4

Next thing you know some of your replicants grow old and decide to kill you, as when an app tries to read the preference it gets an unexpected result:

#!/usr/bin/python

from __future__ import unicode_literals
from __future__ import print_function
from Foundation import CFPreferencesCopyAppValue

print(CFPreferencesCopyAppValue("lifespan", "com.tyrell.nexussix")) # sometimes None, sometimes 4

Yet if we look with the defaults command, everything seems fine:

# defaults read /Library/Preferences/com.tyrell.nexussix
{
    lifespan = 4;
}

To make things even more confusing, CFPreferences suddenly sees the value now. To untangle this mess, let’s investigate what happens in our postinstall script when we’re installing the package on the current system, with $3 expanded to /:

# defaults write //Library/Preferences/com.tyrell.nexussix lifespan -int 4
# defaults write /Library/Preferences/com.tyrell.nexusseven lifespan -int 4
# ls -lf /Library/Preferences/com.tyrell.nexus*
-rw-------  1 root  wheel  57 30 Jan 12:38 /Library/Preferences/com.tyrell.nexussix.plist
-rw-r--r--  1 root  wheel  57 30 Jan 12:38 /Library/Preferences/com.tyrell.nexusseven.plist

The extra slash causes defaults to make the preference file readable only by root, making it sometimes visible through cfprefsd, depending on access permissions and caching. My guess is the slash triggers defaults’ codepath for writing to user libraries, where preference files shouldn’t be readable by other users.

So how should we be writing our postinstall scripts? I’m glad you asked. I’ve been using the following in my scripts:

#!/bin/bash

if [[ "$3" == "/" ]]; then
    TARGET=""
else
    TARGET="$3"
fi

defaults write "$TARGET/Library/Preferences/com.tyrell.nexussix" lifespan -int 4

So what happens if you install this package on a system image with AutoDMG? Well the pref plist will only be readable by root, but as far as I can tell apps can still access their preference keys, presumably since cfprefsd caches everything when the system boots — at least I haven’t been able to reproduce the problem this way.

References:

AutoDMG v1.8

2017-09-26T00:00:00+00:00

10.13 was released yesterday, and today I’m releasing the final build of AutoDMG 1.8 with full support for building both APFS and HFS+ images. Apple has made a lot of breaking changes in this cycle, to the point where we haven’t been able to get 100% support for 10.13 on launch day in my organization. As a result I’ve had to bump a few outstanding AutoDMG issues to the next release, but they mainly concern usability and shouldn’t stop anyone from using 1.8.

This release also means that 10.9 support is dropped, and 10.10 is marked as deprecated. AutoDMG will launch and you can build 10.10 images, but there are no new updates from Apple.

Full changelog:

Added 10.13 and APFS support.
Added APFS/HFS+ selection to the Advanced Options window.
Added -F/--filesystem option to CLI.
Added Filesystem option to templates.
Removed 10.9 support.
Fixed crash when CLI build can’t eject image.
Fixed crash when unchecking updates without a source.
Fixed Unicode handling of paths for additional packages.
Slightly improved app launch time.

AutoDMG v1.7.3

2016-12-19T00:00:00+00:00

This week I’ve rounded up some bugfixes together with a few minor new features. First off, after a successful build you’ll find a few machine parsable statements in the log, to help people who automate builds. You’ll also see a warning if the build number changes, as it does these days when you install a security update:

december 2016 11:35:55 GMT−5  Build success: OUTPUT_PATH='/Users/Shared/osx_updated_161216-10.11.6-15G31.hfs.dmg'
december 2016 11:35:55 GMT−5  Build success: OUTPUT_OSNAME='Mac OS X'
december 2016 11:35:55 GMT−5  Build success: OUTPUT_OSVERSION='10.11.6'
december 2016 11:35:55 GMT−5  Build success: OUTPUT_OSBUILD='15G1212'
december 2016 11:35:55 GMT−5  Build success: Notice: OS version changed from Mac OS X 10.11.6 15G31 to Mac OS X 10.11.6 15G1212

Secondly I’ve added a deprecation warning if you’re building an OS that is no longer supported by Apple, as is the case now with 10.9. This also means that I will remove support for it in AutoDMG in the not too distant future, most likely in the next major release.

Full changelog:

Display warning if Apple has stopped updating the target OS.
Only update the profile if the running OS is supported.
Handle already mounted InstallESD.dmg if installer app was moved afterwards.
Abort the build if disk image creation fails.
Print warning in log when build number changes during install.

AutoDMG v1.7.2

2016-12-19T00:00:00+00:00

1.7.2 was pulled due to a crash on startup for new users, please use 1.7.3 instead.

AutoDMG v1.7.1

2016-12-08T00:00:00+00:00

Hot on the heels of last week’s release, here’s AutoDMG 1.7.1 where I’ve focused on polish and bugfixing. This release brings the list of open issues down to zero, which feels really good! Unfortunately that also meant closing a few older feature requests that I wanted to implement, but when push came to shove if there’s a simple one-line workaround in the Terminal, it’s hard to motivate spending half a day on implementation and regression testing. Here’s what did make the cut:

Now displays the current OS name and placeholder icon.
New menu item to select an installer, ⌘I. ⌘⌥I will pick the first installer it finds in /Applications.
Log window wraps long lines and shows colorful warnings.
Preserve TMPDIR when running CLI builds without sudo.
Better error handling when ejecting the temporary build image.

CreateUserPkg is up for adoption

2016-12-07T00:00:00+00:00

As some of you might know, I’ve been on a leave of absence this autumn, and I’ve basically been fortunate enough to have the days to myself. I’ve spent most of it on bike rides around Columbus, playing around with the laser cutter at the Columbus Idea Foundry, or preparing for the talk I gave at MacTech, but over these last couple of weeks I’ve been catching up on some of my open source projects.

Even with all this copious free time, I’ve had to realize that I have more projects than I have time for, and when January comes around I’ll be back to working full time again. The project that is in the most dire need of an overhaul is CreateUserPkg. It was one of my first Cocoa projects, where I learned how to do document based apps, sandboxing, and how to publish in the App Store. I had to resort to some really crazy antics to generate packages without requiring root. Good times. But that was then, this is now, and a lot has changed:

macOS ships with pkgbuild and productbuild, replacing the need for most of the horror show in create_package.py.
10.5/10.6 support is no longer needed, so the SHA1 shadow hash needs to be eliminated in favor of PBKDF2.
Using .pkg as the document type was just a bad design desicion, I should have made a separate document format and then exported to .pkg.
MDM payloads can create users, maybe we should be generating those instead?
The app desperately needs a better name too. I’ve always been bad at naming things. Fortunately Greg saved my bacon with AutoDMG.

On top of it all the project is small enough that it’s begging for a Swift rewrite. But not by me. If you’re looking for an excuse to get into Cocoa development, here’s a great opportunity. There are a dozen suggested enhancements on GitHub you want to go further.

So please, fork the repo, or maybe just start over from a clean slate. The Apache license means that you can basically do what you want with the code. The only thing I request is that you give your app a better name.

AutoDMG v1.7

2016-12-01T00:00:00+00:00

I was planning a small bugfix release, but I felt that the changes were large enough for a major version bump. The AppleScript trampoline that I used to kick off the privileged OS install task has been replaced by the much nicer STPrivilegedTask. If you have automation set up to run AutoDMG, please test this release carefully.

Rewrote launch code for the privileged task that runs the OS installer.
Fixed beachballing during finalize task.
Fixed language detection for Setup Assistant on 10.12.

AutoDMG v1.6

2016-09-19T00:00:00+00:00

Just in time for macOS Sierra we have AutoDMG 1.6:

Added 10.12 support.
Removed 10.8 support.
AutoDMG will not try to remount already mounted images, fixing 10.12 installer.
Added advanced option to skip asr imagescan (thanks @timsutton).
AutoDMG will not relaunch on login (thanks @nmcspadden).
Fixed volume size setting not being saved to templates (thanks @timsutton).
Fixed handling of relative package paths on the command line (thanks @timsutton).
Fixed usage of base image as source on the command line (thanks @timsutton).

You can download 1.6 from the AutoDMG releases page. If you have questions or need help, you can join us in #autodmg on Slack MacAdmins (but please check the documentation and especially the FAQ first).

AutoDMG v1.5.5

2016-01-22T00:00:00+00:00

I released AutoDMG 1.5.5 with a few bug fixes:

Respect TMPDIR when building images on 10.9+.
Avoid occasional error messages on program exit.
Fixed handling of packages where size can’t be calculated.

You can download 1.5.5 from the AutoDMG releases page. If you have questions or need help, you can join us in #autodmg on Slack MacAdmins (but please check the documentation and especially the FAQ first).

AutoDMG v1.5.4

2015-11-27T00:00:00+00:00

Some of the changes I’ve planned for AutoDMG 1.6 are taking way too long, so after some gentle nudging I rolled up the latest bug fixes into a 1.5.4 release:

Added ⌘B keyboard shortcut to build (thanks @zuhl).
Added -s/--size option to CLI (thanks @timsutton).
Fixed error handling of bad Additional Packages paths.
Display confirmation dialog when updating profiles.
Disable profile update checking while images build.
Fixed handling of large error messages from the build script.

You can download 1.5.4 from the AutoDMG releases page. If you have questions or need help, you can join us in #autodmg on Slack MacAdmins (but please check the documentation and especially the FAQ first).

AutoDSNBI 1.1

2015-11-03T00:00:00+00:00

I updated AutoDSNBI to support DeployStudio 1.6.17. Big thanks to Rich Trouton for pointing out the new sysbuilder script. Download 1.1 from the new repo here:

AutoDSNBI 1.1

System Center 2012 Endpoint Protection + Outlook 2011 = Crash

2015-08-17T00:00:00+00:00

Starting in July we’ve had a steady trickle of support cases regarding Outlook 2011 crashing on startup:

Microsoft Error Reporting log version: 2.0

Error Signature:
Exception: EXC_BAD_ACCESS
Date/Time: 2015-08-11 11:38:17 +0000
Application Name: Microsoft Database Daemon
Application Bundle ID: com.microsoft.outlook.databasedaemon
Application Signature: MDlr
Application Version: 14.5.3.150624
Crashed Module Name: libobjc.A.dylib
Crashed Module Version: unknown
Crashed Module Offset: 0x000010a7
Blame Module Name: Microsoft Database Daemon
Blame Module Version: 14.5.3.150624
Blame Module Offset: 0x000c1368
Application LCID: 1053
Extra app info: Reg=en Loc=0x041d
Crashed thread: 0

We tried the usual incantations — rebuilding the database, creating a fresh identity, nuking prefs, and so on — but nothing offered more than temporary respite. Only creating a fresh home directory seemed to help, but that’s a classic example of the cure being worse than the disease.

We’re fortunate enough to have a Microsoft Premier Support contract so after banging our heads against the wall for a few days we decided to call it in. Impressively they managed to track down the issue in less than 24 hours. It turns out that a recent-ish update to System Center 2012 Endpoint Protection accidentally changes the ownership of ~/Library/Preferences/loginwindow.plist to root:wheel. Through a series of Rube Goldberg-like events this then causes Outlook’s database daemon to crash on startup.

Restoring ownership magically makes the crash go away, so we sent out the following script to all our macs:

#!/bin/bash

# Restore ownership of user preference files.
for preffolder in /Users/*/Library/Preferences; do
    if [[ -d "$preffolder" ]]; then
        uidgid=$(stat -f "%u:%g" "$preffolder")
        chown -hR "$uidgid" "$preffolder"
    fi
done

I think SCEP is just a rebranded ESET/NOD32, so chances are that the same problem can affect clients running that too.

I haven’t been able to reproduce the issue by just changing loginwindow’s ownership on a non-crashy Outlook, so something else is also playing a part here, but at least it fixes the cases that have been brought it. Don’t hesitate contact me if you can shed more light on it.

Ultima IV Remastered v2.2.1

2015-05-29T00:00:00+00:00

I apparently managed to screw up the end game graphics while preparing the 2.2 release, which Trurl brought to my attention. Sorry folks!

What’s New in v2.2.1

Fixed graphics corruption and potential crash during ending.

For older changes see the full changelog.

Download & Source Code

You can download the game from its CSDb entry: Ultima IV Remastered v2.2.1.
The source code is available on GitHub.

Ultima IV Remastered v2.2

2015-04-22T00:00:00+00:00

A few more bugs were reported (thank you!), and I really wanted to get my list of known bugs down to zero. Unless people discover even more bugs this is likely to be the final release.

Atari & Apple II ports

Users from the other 6502 platforms have also shown some interest in remastered versions. I don’t have any contacts in the Apple or Atari cracking scene, so I thought I’d put a note in here. If you’re interested let me know and I can help you navigate the project.

What’s New in v2.2

The game no longer hangs when playing the 1541 version and answering questions wrong in the codex.
Attacking static NPCs such as Ankhs, castle lords & ladies, the water in Castle Britannia, or the camp fire in Yew no longer leads to a glitching enemy in combat. Attacking chests dropped by townespeople is no longer possible.
Fixed stack handling for main menu, when being attacked by monsters, when dying, and the end game. This hopefully eliminates any crashes to the ready prompt.
The avoid combat trainer now works in dungeons too.

For older changes see the full changelog.

Download & Source Code

You can download the game from its CSDb entry: Ultima IV Remastered v2.2.1.
The source code is available on GitHub.

Ultima IV Remastered on GitHub

2015-04-12T00:00:00+00:00

One of the primary reasons for getting back into Ultima IV was that I wanted to clean up my old Ultima IV Gold sources. Remastering was the result of sliding down that slippery slope of just fixing one more thing.

The restarted project was born out of nice, clean, structured source but when crunch time came ideals were abanonded and compromises were made. Sadly the project isn’t in as nice a shape as I’d like and there are some nasty gotchas in there, but if I don’t release it now I risk having the project rot away in my private repo again, and bringing us back to square one. You can check out the source, warts and all, from u4remastered on GitHub.

For C64 programmers there are some reusable parts like the IFFL and EasyFlash loaders and LZMV compression (of which I have more, in the repo is just the pack tool and the stream decruncher).

My work should also be portable to the Apple II and Atari versions if anyone is up for it, and the dialogue fixes should be adaptable to the DOS/Amiga/ST versions as well.

Finally, for the curious I’ll be keeping continued bugfixing and development on GitHub if you want to follow along. I’ll start moving my todo list into the project’s issue tracker later today.

Ultima IV Dialogue

2015-04-10T00:00:00+00:00

One step on the road to remastering Ultima IV was extracting all dialogue from various ports and looking at the differences. I have published the json dumps, my tools, a comparision, and a full changelog on GitHub.

Ultima IV Remastered v2.1

2015-04-09T00:00:00+00:00

Ultima IV Remastered v2.1

I had a deadline set for the release of Ultima IV Remastered, which really helped me finish the project. Unfortunately it also meant that I had to release it with a few known bugs, and, as it turns out, a few unknown bugs too.

A big thank you to all who sent me bug reports and posted feedback, keep it coming!

What’s New in v2.1

Saving to EasyFlash cartridges that don’t use Am29F040 chips (including Chameleon) now works.
Exit solo mode if character leaves combat.
Don’t spawn a ship when pressing B while standing on a ladder in a dungeon.
Joshua courage -> canrage typo.
Fixed east/west start positions in Destard rooms on level 8.
Removed glitch when attacking sick people in townes.
Play correct music after dying and being resurrected by Lord British.
Fixed wind status after loading a game in a dungeon and then exiting.
Clarified the recommendation for emulator users when flipping d64s.
Proper error message in editor if no savegame is found in cartridge.

Download

You can download the game from its CSDb entry: Ultima IV Remastered v2.2.1.

Known Bugs

If you’re playing the 1541 version and answer a question wrong during the ending the game is supposed to throw you out to Britannia, but instead it hangs. Play the 1571, 1581, or EasyFlash version instead to avoid this issue, or just make sure you get all the answers right.
The crack intro is out of sync and glitches a bit on NTSC, but should otherwise work fine.

Ultima IV Dungeon Ladder Frigates

2015-04-02T00:00:00+00:00

I released Ultima IV Remastered to great fanfare last weekend and it warms my heart to hear that you’re all enjoying it. I just received a bug report though that I thought was neat enough to share. As Codetsu reported on Lemon64 if you press the B key while standing by a ladder going up in a dungeon, the game rather unexpectedly responds with “Board frigate!” (screenshot from the original Ultima IV port):

I don't see a frigate. Do you see a frigate?

The game continues to play normally but you get a nasty surprise as you exit the dungeon:

Oh there it is!

You can exit the ship and walk away from the dungeon, but reentering is impossible until you move far enough away (roughly 30 steps, or use the teleport cheat) that the game’s object buffer is cleared for that part of the map. Unless you’re exiting Covetous, in which case you can just sail off since there is water right next to the dungeon entrance - a fairly handy way to spawn a ship in the original game with no cheats. But I digress.

What’s really happening here though? Let’s look at the game’s code. The raw disassembly is a little tough to read so I’ve added labels:

; Disassembly starting at $4769:

cmd_board:
	lda player_transport        ; This is $00, so we branch to @onfoot.
	cmp #$1f
	beq @onfoot
	jsr j_primm
	.byte "Board <-", $00

	jmp print_cant

@onfoot:
	lda tile_under_player       ; If the player is next to a ladder going up
	cmp #$14                    ; the current tile is $10.
	beq board_horse
	cmp #$15
	beq board_horse
	jsr j_primm                 ; It's not a horse so the game prints "Board ".
	.byte "Board ", $00

	lda tile_under_player
	cmp #$10                    ; As it happens the ladder up dungeon tile is
	beq @ship                   ; the same as a ship facing left on the
	cmp #$11                    ; surface, $10, so we branch to @ship.
	beq @ship
	cmp #$12
	beq @ship
	cmp #$13
	beq @ship
	cmp #$18
	bne @unknown
	jmp board_balloon

@unknown:
	jmp cmd_unknown

@ship:
	lda #$10                    ; Now it tries to remove $10 from the object
	jsr board_find_object       ; table. No object is found, but the current
	jsr j_primm                 ; player_transport is still set to $10 by
	.byte "frigate!", $8d       ; board_find_object (see below).
	.byte $00

	lda player_xpos             ; A new ship is spawned facing left at the
	cmp player_ship_xpos        ; player's current surface coordinate.
	bne @newship
	lda player_ypos
	cmp player_ship_ypos
	bne @newship
	jmp cmd_done

@newship:
	lda #$50
	sta ship_hull
	jmp cmd_done


; Disassembly starting at $480b:

board_find_object:
	sta board_object_id
	ldx #$1f
@checkobject:
	lda object_tile,x           ; The object_tile table doesn't contain any
	and #$fc                    ; ships, so this compare will always fail.
	cmp board_object_id
	bne @next
	lda object_xpos,x
	cmp player_xpos
	bne @next
	lda object_ypos,x
	cmp player_ypos
	beq @foundobject
@next:
	dex
	cpx #$08
	bcs @checkobject
	bcc @noobject
@foundobject:
	lda #$00
	sta object_tile,x
	sta map_status,x
@noobject:
	lda tile_under_player       ; After exhausting the table it ends up here,
	sta player_transport        ; where it still updates player_transport to
	rts                         ; whatever is on the map under the player.

What’s the actual bug here? IMHO the game shouldn’t assume that just because the player is on foot, it could board something — it also needs to check that the player is on the surface. The cleanest way to do so would be at the very beginning, but when you insert a binary patch you don’t want to add or remove any bytes to the program, as that breaks everything coming afterwards. It’s best if we can find a jmp or jsr that can be changed to point to a patched routine. Following the code we see that no game state is being changed before it jumps to the board_find_object subroutine, so let’s redirect that:

patch_board_dungeon:
	.byte 2                     ; Patch two bytes.
	.addr $47b1                 ; Address to patch.
	.addr board_ship_check      ; The new vector.
	.byte 0                     ; End of patch.

I’ve added a short routine to the game’s startup that takes a table with patches such as this, making it relatively painless to modify the game without having to recreate the whole source. Now we can insert a small stub with a surface check:

board_ship_check:
	lda current_location        ; Location $00 is Britannia surface.
	beq @in_britannia           ; If zero, continue as normal.

	pla                         ; Otherwise we pop the jsr return address and
	pla                         ; jump to printing "what?", just as if no
	jmp cmd_unknown             ; object was found.

@in_britannia:
	jmp board_find_object

Let’s compile the project and see how this plays out:

No more phantom ships hanging around dungeon ladders.

With that bug squashed and the code commited to a new bugfix branch it’s time for regression testing. Since we just changed the code for the board command, let’s make sure we didn’t add any new board related bugs. Pop up to the surface, spawn a horse, a ship, and a ballon, and try to board them.

I only spawned one ship, so why are there two?

Boarding a ship and sailing away leaves another copy of the ship on the map. With larger changes hunting down regressions can be a pretty daunting task, but with a change this small it’s bound to be something stupid. And it is: when board_find_object is called, the object to find is in the A register, but the first thing we do in our patch is clobbering A with current_location. Let’s change it to X which isn’t used when calling board_find_object:

board_ship_check:
	ldx current_location        ; Location $00 is Britannia surface.
	beq @in_britannia           ; If zero, continue as normal.

	pla                         ; Otherwise we pop the jsr return address and
	pla                         ; jump to printing "what?", just as if no
	jmp cmd_unknown             ; object was found.

@in_britannia:
	jmp board_find_object

Compile, re-test, and this time there are no obvious regressions:

With a game the size of Ultima IV there are dozens of little bugs like this that weren’t caught before release. To the best of my knowledge there were no tools for testing 6502 code in 1985 and even revision control systems were rare, so it’s impressive that there aren’t more of them. Not counting conversation bugs this is the seventh bug I’ve fixed, and out of them only the shrine bug has any real effect on the game — even so it’s fairly minor and was patched in a later update.

Anyway, the branch is merged and the fix will be included in the next update of Ultima IV Remastered. Let me know if you find any other bugs!

Ultima IV Remastered

2015-03-30T00:00:00+00:00

I’m proud to announce that my remastered port of Ultima IV for the C64 has been released.

Ultima IV Remastered for the C64

Download

You can download the game from its CSDb entry: Ultima IV Remastered v2.2.1.

Documentation

Scanned documentation can be found here: Ultima IV Documentation.

Screenshots

You can download screenshots of U4 Remastered and the original for comparison here: Ultima IV Remastered screenshots.

Source Code

The source code is available here.

Game Changes

Graphics Overhaul

Character creation intro has new graphics drawn by Vanja ‘Mermaid’ Utne.
The tileset has been updated with new and more colorful tiles, mainly backported from Ultima V.
When peering at a gem or casting view the map is now in color.
The dungeon renderer has been sped up, and items and monsters are drawn in color.

Text Overhaul

NPC dialogue and game text now uses mixed case.
Many dialogue bugs have been fixed:
- Fixed broken question trigger and added missing dialogue for Thevel in Britain.
- Added missing dialogue for Serpent’s Hold gate guards.
- Changed keyword for Michelle to avoid conflict with health in Serpent’s Hold.
- Fixed broken message trigger for Water in Castle British.
- Fixed misspelled keyword for Estro in the Lycaeum.
- Fixed misspelled keyword for a poor beggar in Yew.
- Fixed broken message trigger for Alkerion in Minoc.
- Fixed broken message trigger for Shamino in Skara Brae.
- Fixed broken message trigger for Charm in Cove.
- Dozens of minor fixes to spelling, grammar, and formatting.

New Game Features

Select active character in combat with 1-8, 0 returns to party mode.
You can quit and save in dungeons.
If you move outside the border of a towne, castle, or keep, the game asks if you want to exit to Britannia.
Joystick control.
Dungeons get darker when a torch is close to burning out.

Bug Fixes

Meditating at shrines now gives different hints for 1, 2, and 3 cycles.
Hythloth dungeon rooms on level 6 are now accessible.
Character creation no longer suffers from random hangs.
Loading a saved game restores balloon flying mode.
Allow backspace when giving gold or using stones.
Don’t enter locations when flying in balloon.

Technical Enhancements

Fastloader with support for 1541, 1571, 1581, CMD FD, and CMD HD.
Runs from a single disk, no disk swapping when playing.
EasyFlash cartridge version.
Music is not interrupted when data is loaded from disk.
SuperCPU support.

Game Cheats

A save game editor is bundled with the game.
Unlimited magic.
Unlimited food.
Unlimited torches.
Unlimited keys.
Unlimited gems.
Avoid combat.
Control balloon.
Idle without pass.
Teleportation system, press T in Britannia and then (T)owne, (D)ungeon, (S)hrine, (L)ocation, or (C)oordinate.
Create transport if you try to (B)oard an empty tile, select (H)orse, (S)hip, or (B)alloon.

Additional Changes Since Ultima IV Gold

Disk drive no longer spins when there is no disk activity, and the game loads faster.
Balloon cheat has several bugs fixed.
Swapped lat/long for teleport cheat.
Added unlimited gems and idle cheats.
Fixed reagent editing in save game editor.

Known Bugs

If you’re playing the 1541 version and answer a question wrong during the ending the game is supposed to throw you out to Britannia, but instead it hangs. Play the 1571, 1581, or EasyFlash version instead to avoid this issue, or just make sure you get all the answers right.
The crack intro is out of sync and glitches a bit on NTSC, but should otherwise work fine.
Blue border doesn’t get redrawn properly around wind status after you exit a dungeon.
If you die and teleport back to Lord British the wrong music plays in Castle Britannia, until you klimb down the ladder or talk to Lord British.
If the savegame editor is started from cartridge and has problems accessing the flash saves you get an error asking you to insert a disk in device #239.

Ultima IV Remastered (teaser trailer)

2015-03-28T00:00:00+00:00

My development projects are usually more or less in the open. As soon as I have something that might be of interest to others I tend to publish the repo on GitHub, where it either sinks or swims. For the last couple of months though I’ve been working quietly on my own. I have dropped a couple of screenshots but I don’t think anyone noticed… As I’ve been itching to talk about it it’s with great relief that I’m finally able to announce:

Ultima IV Remastered for the C64

Ultima IV has a special place in my heart.

Back in 1987 when I bought my first computer, I started trading games on tape with my friends, and dabbled a bit in basic and machine code programming. But after my friend visited his uncle he came back raving about this marvelous game that let you walk around on a world map, explore dungeons, and talk to people in towns and castles. The game would even save your progress so you could continue the next day. The only snag was that you needed a disk drive to play it, which none of us had.

It took me another year before I could save up enough money to buy a second hand disk drive, but when I did the first game I bought was Ultima IV. I spent countless hours exploring Britannia, reading through the faux leather cover manuals, and once I learned how to access raw sectors from the disk drive I started decoding the game’s data to print maps on the screen.

My home away from home.

Later on I took a proper look at disassembling the game and figuring out how it worked. I ended up rewriting the game’s loader to work on larger disk drives, and fixed the game so it didn’t run too fast with a SuperCPU accelerator. After about four years worth of spare time it was released in 2006 as Ultima IV Gold. My intention was always to clean up the source a bit and release an updated version, but as with most projects it ended up sitting in a backup archive of my old Windows PC instead. My development environment slowly mutated, as it tends to do, so the archive turned into an uncompilable mess of old Perl scripts and jumbled Makefiles that somehow made sense at the time, but only now only produces a fountain of error messages.

☥

Some recent events in the C64 scene made a few people ask if maybe the Ultima IV Gold loader was available, but it saddened me to have to tell them that I couldn’t even get it to work myself. This, combined with a week off work, made me decide that I might as well take a look at the mess. In the end it only took about a day’s work, and after that I was hooked again. With a fresh Makefile written from scratch (if you’re a programmer and haven’t read Recursive Make Considered Harmful, go do so immediately after this) and all the support tools rewritten in Python the project came back to life.

I made a little research and compiled what I think is a fairly complete list of bugs in the 6502 ports (the C64 and Atari 800 versions share over 90% of the source code and data with the Apple II original), and it was a joy to hunt down every last one of them. The DOS version has received several patches too, and since the file formats are virtually identical I could backport those patches too.

This room can't be reached in the original.

The largest job though was fixing the dialogue, simply because there is so much of it. The early releases were littered with typos and downright bugs where some paths of the dialogue were inaccessible. Fortunately it got progressively better with each port and the last DOS version provided a good starting point. Interestingly the 68k ports were independently fixed and differ both in style and how the bugs have been fixed.

These guards used to reply with Sentri's dialogue.

While Chuck Bueche did a good job porting Ultima III and IV to the C64, not least with a great interpretation of Ken Arnold’s music, he didn’t use the hardware to its fullest potential. For Ultima V they hired David Shapiro who was a lot more experienced with the platform - not that it’s fair to compare with the earlier releases, there is a world of difference between C64 games in 1985 and 1988.

Disregarding the massive leaps forward that the core game engine did in U5, the most obvious change is probably that of the graphical richness that Shapiro managed to bring to the game, considering that it uses the same graphics mode as U3 and U4. The thought struck me that if I could just find 0x300 bytes for the extra color tables there’s no reason U4 couldn’t look just as good. Since memory was already full in my 2006 release it would need a lot of restructuring, as the color tables need unbroken pages of 0x100 bytes and there were only scraps here and there. It’s always a little unnerving to rearrange memory when you’re working with someone else’s code, since what looks like free space might be used at a later point in the game, but by relocating some of my own patch code to gaps in lower memory I could transplant the tileset and color tables from Ultima V. It’s impossible to port the full tileset, since the engine in Ultima V can handle 512 tiles and Ultima IV is limited to 256, but it still brings a marked improvement.

Ultima IV on the Apple II, the original C64 port, and my remastered version.

I also decided to backport one of the best improvements in the U5 engine, where you can select an active character in combat — this makes a world of difference when exploring dungeon rooms.

Of course, after replacing the tileset the character creation intro graphics really sticks out like a sore thumb. The original port has a very primitive conversion of Marsha Meuse’s Apple II graphics, and the C64 is capable of so much better. I checked with my friends in Genesis Project and to my great joy Vanja ‘Mermaid’ Utne was interested in taking on the massive job of redrawing the intro graphics. The difference is, to say the least, stunning.

The original intro graphics, and Vanja's replacement (based on the FM Towns version).

Since I was on a roll I figured that I might as well fix one of my pet peeves with the game. For something that relies so heavily on telling a story with text, it has always baffled me that EVERYONE SHOUTS when you talk to them in the 6502 ports. The engine is perfectly capable of printing lower case characters, and it’s used everywhere else. Since I had already extracted all the dialogue as part of the earlier bug hunt it was an easy job to port a cleaned up version of the DOS dialogue back in. A minor change in string encoding also meant that I could free up 16 bytes in every conversation, allowing me to put back some punctuation that had to be cut due to a lack of the space in the original.

☥

Of course, it didn’t end there, but this blog post is already way too long. For the full details you’ll have to wait until it’s released at Gubbdata this weekend. All told, I’ve had a great time revisiting this old project and I can’t wait to share it with the world!

C64s packed and ready to leave!

AutoDMG v1.5.3

2015-03-24T00:00:00+00:00

I have released AutoDMG 1.5.3 which contains the following bugfixes:

Include custom volume name when saving a template from the GUI.
Fixed resolving of certain aliases on 10.10.
Fixed error message when log file couldn’t be opened.
Fixed building when a mobile account is roaming and can’t look up its username.

You can download 1.5.3 from the AutoDMG releases page. If you have any questions join us in ##osx-server (but please check the documentation and especially the FAQ first).

AutoDMG v1.5.2

2014-11-13T00:00:00+00:00

AutoDMG 1.5.2 has been released with a small bugfix:

Fixed Setup Assistant country/language selection on 10.10.

You can download 1.5.2 from the AutoDMG releases page. If you have any questions join us in ##osx-server (but please check the documentation and especially the FAQ first).

AutoDMG v1.5.1

2014-10-31T00:00:00+00:00

AutoDMG 1.5.1 has been released with a small bugfix:

Fixed crash on startup when ~/Library/Logs has wrong permissions.

You can download 1.5.1 from the AutoDMG releases page. If you have any questions join us in ##osx-server (but please check the documentation and especially the FAQ first).

AutoDMG v1.5

2014-10-16T00:00:00+00:00

Apple just released 10.10, and here’s AutoDMG to go with it. Changelog is nice and short:

Added 10.10 support.
Removed 10.7 support.

Happy Yosemite day!

You can download 1.5 from the AutoDMG releases page. If you have any questions join us in ##osx-server (but please check the documentation and especially the FAQ first).

AutoDMG 10.8 maintainer needed

2014-10-15T00:00:00+00:00

Since I no longer support 10.8 at work I will not be able to stay on top of updates for AutoDMG. With the upcoming release of AutoDMG 1.5 I’ve removed support for 10.7 (stay on 1.4.4 to build 10.7 images), and unless 10.8 is to suffer the same fate someone needs to step up. It’s not particularly hard: fork AutoDMGUpdateProfiles, follow the suggested workflow and send pull requests on github.

As always I will do my best to keep up with the latest release of OS X, and the one before, but pull requests are most welcome there too.

Also, I’ve uploaded a new beta of AutoDMG 1.5. I expect this build to become the final release version and I’m just waiting for the final GM to drop.

AutoDMG v1.4.4 and 1.5b

2014-08-20T00:00:00+00:00

Thanks to our great community a few more AutoDMG bugs have been discovered. I rolled up the fixes today and released 1.4.4:

Fixed crash when user has unicode characters in primary group name.
Better error handling when downloads fail or additional packages are moved.
Verify passwords in CLI to avoid failed builds.

I’ve also released a new beta of 1.5 for Yosemite testers. It contains the same fixes as 1.4.4.

You can download 1.4.4 and 1.5b from the AutoDMG releases page. If you have any questions join us in ##osx-server (but please check the documentation and especially the FAQ first).

Swift Impressions

2014-06-05T00:00:00+00:00

Of all the announcements at WWDC the one that got me the most excited was that of Swift, the new programming language. I looked at Go and Rust earlier this year, and while I find them interesting I didn’t find a compelling reason to switch from Python, my trusty go-to language for the past few years. And even though I prefer Python for most projects, when you’re developing Cocoa applications you always feel like a second-class citizen. The crazy long method names are a small but ever-present annoyance, and I’ve spent way more time than I like trying to herd runaway exceptions in background threads.

So I’ve been on the fence lately, thinking that maybe I should trade some of Python’s ease of development for Objective-C’s promise of a more stable code base down the road. Thus, the Swift announcement couldn’t have come at a better time. The stability and performance of a modern, type safe language, together with the automatic memory management and flexible container types that Python has spoiled me with. What’s not to love?

I took the plunge and installed the Xcode 6 beta, and promptly lost a day futzing around with Storyboards and Auto Layout - I really need to take the time to learn that stuff. But then I spent most of yesterday making a silly little app to display the status of the Absolute Manage agent and what packages it has queued for install, giving Swift a proper workout. My impressions:

The Good

The syntax is nice and consistent, with no obvious warts.
All the things you’d expect from a modern language, like type inference and closures.
Optionals provide an elegant and safe alternative to NULL pointers.
It’s refreshing to actually have to read the documentation, since there’s virtually nothing of substance on Google or StackOverflow.

The Bad

It’s frustrating to actually have to read the documentation, since there’s virtually nothing of substance on Google or StackOverflow.
There’s no way to catch exceptions, yet we’re expected to call classes that can raise them.
I’m missing list comprehensions.

The Ugly

The Xcode syntax highlighter keeps crashing and restarting.
I’ve managed to find no fewer than two bugs with array downcasting that causes the compiler to crash.
Error messages are rarely helpful:

AMSController.swift:52:38: error: cannot convert the expression's type 'AnyObject[]!' to type 'AnyObject[]!'
if let urls = fm.contentsOfDirectoryAtURL(dirURL, includingPropertiesForKeys: [], options: 0, error: &error) {
              ~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

(I think options: 0 should be options: nil since in Swift it’s a struct and not a bitmask - at least that makes the compiler happy.)

Verdict

Overall the first impression is overwhelmingly positive, and whatever my next Cocoa project will be I’m certain that I’ll be writing it in Swift. The only problem is that the toolchain is currently too buggy for actual work, but I’m sure that will improve in the next couple of betas.

DS NetBoot sets from the command line

2014-05-27T00:00:00+00:00

If you, like me, only build DeployStudio NetBoot sets once in a blue moon I’m guessing you also tend to forget all the configuration options in the assistant and have to look them up each time. With the torrent of recent DS releases (not that I’m complaining!) I found myself running the assistant one too many times and I figured that there must be a way to automate it. There was a hint in the forum a while back (post #5) and recently PaintedTurtle posted a sample script showing that it’s actually pretty straightforward. I wrapped it up in a script with configuration options in the same order as the assistant to keep it nice and simple:

AutoDSNBI

As a bonus it accepts either a system volume or a system image, e.g. your latest AutoDMG image.

I’ve built a couple of NBIs with it now but I haven’t tested it extensively, so caveat emptor.

AutoDMG v1.4.1

2014-05-15T00:00:00+00:00

AutoDMG 1.4.1 is out:

Added support for variables in templates (CLI only).
Fixed a crash when trying to mount certain dmgs.

Thanks to Anthony Reimer for finding the dmg bug!

You can download 1.4.1 from the AutoDMG releases page. If you have any questions join us in ##osx-server (but please check the documentation and especially the FAQ first).

Implementing A* in 6502 assembler

2014-05-07T00:00:00+00:00

Last time I implemented a priority queue using a heap, and this time I’m going to use it as the basis of implementing A* pathfinding for my RPG (1 2). As always the code is available on GitHub: AsmAstar.

I’m not going to try to explain the algorithm here, but I’m going to focus on how I implemented it and the optimizations I chose. I recommend reading Amit Patel’s Introduction to A* as it’s the best article on A* I’ve found. Update: Amit wrote a new (interactive!) Introduction to A*.

Playing with A*

This time I also started with a high level implementation to get the details right before writing low level code, and I’d been looking for an excuse to play with Pythonista. I can’t say I enjoyed writing code on my iPhone though, a Bluetooth keyboard is a must before I try that again. Anyway, I shared the code as a gist if you want to play with it it:

Pythonista code: Astar.py

The core of the algorithm is in find(). Since writing code on my iPhone drained my patience it’s missing a few vital optimizations, making it perform quite badly. I can always claim that it’s slow so you can watch as the algorithm expands its search tree.

The open set

A* spends most of its time inserting and deleting nodes from the open set, which is why it’s important to choose the right data structure for it. Depending on what kinds of maps it’s traversing the ratio of insertions to deletions is going to vary, but it’s likely to be heavy on insertions. In my game, with 8 directions of movement in rooms that rarely have more than 10x10 tiles of open space, I’ve found that it inserts about twice as many nodes as it deletes. Fortunately the heap that I implemented in my last article has O(log n) complexity and is slightly faster at inserting than deleting, fitting the task nicely. It only needs to be widened so that instead of just maintaining a tree of single bytes it has parallel trees of priority, cost, xpos, and ypos that are all manipulated in tandem.

The closed set

A* also needs to keep track of nodes that have already been explored, the set of closed nodes. To support large maps this needs to be a dynamic data structure that features fast lookups and doesn’t use too much RAM.

One vital distinction that needs to be made is that nodes in the A* search tree are not the same as map coordinates. The same coordinates can be visited by multiple nodes when alternative paths are explored. There’s an optimization opportunity here though: only insert nodes for a coordinate if the cost is lower than that of previously visiting nodes. This has a large impact on the number of nodes that are inserted. Also, when grabbing a node from the open set, if the coordinate has been visited previously by a less costly node, there’s no need to examine it as all its children will be more expensive. Taken together they mean that even though there may be multiple nodes for a coordinate in the open set, instead of a set of closed nodes we only have to store a cost for each coordinate, and this can be done with a simple array. There are two values that need to be stored for each coordinate:

The lowest cost that it has been reached by.
Which neighbor the path came from.

Since I allow 8 directions of movement 3 bits is enough to store the neighbor. My action point system uses a cost of 5 for orthogonal movement and 7 for diagonal, but even if I used a cost of 2/3 I would need at least 6 bits to represent path cost. Thus the only option is to use at least two bytes of storage for each coordinate.

Sliding window

This has turned the problem from one of speed or complexity into one of size. I have a 4 kB buffer for map tiles, but there simply isn’t space to allocate another extra 8 kB. Fortunately I don’t have creatures moving across the whole map since combat is limited to one room at a time, and rooms are never larger than 21 x 21 tiles, neatly fitting them on a single screen. This means that I can have a window that slides across the map following the action and only need two 441 byte buffers for cost and direction.

Current room borders (red) and sliding window (green).

From C to assembly

I started out with everything except the heap in plain C and then worked out which functions took the most time. The low hanging fruit was array access to the map, cost, and direction buffers since it involved 16-bit multiplication, and they’re called several times for each node. The other function that was easy to optimize was the A* heuristic calculation (based on Chebyshev distance multiplied by 5 for orthogonal steps and 7 for diagonal steps), called once for each node insertion. Instead of relying on the C compiler’s generic 16-bit math I use hardcoded 8-bit multiplication with shifts and adds, making it an order of magnitude faster.

With the micro optimizations out of the way I started looking at converting the code for examining neighbors and inserting them into the open set. The results were rather disappointing though — despite inlining the cost check/update and using registers instead of the stack to pass arguments the net result was a meagre 5% speedup. In the end keeping the code clear and maintainable is more important than squeezing out the last bit of performance, it’s a turn based game after all.

Sandbox application

I developed a small sandbox application for testing. You can place unpassable boulders on a 21 x 21 map and watch as the algorithm tries to wiggle its way around them. The closed set is visualized with arrows displaying the direction from which it was added. You can download it here:

Prebuilt binary: AsmAstar.d64
Source code: AsmAstar.

Run it with x64sc from the VICE Emulator Suite.

Implementing a priority queue in 6502 assembler

2014-05-02T00:00:00+00:00

I have an old game programming project (obligatory screenshots: 1 2) that I have been playing with off and on for a few years now. It’s a turn based combat RPG for the C64 and the engine is a great playground for testing algorithms and experimenting with data structures. There’s a treasure trove of interesting problems to solve, like realtime field of vision, combat line of sight, and enemy AI. I did an implementation of A* pathfinding in assembler a few years ago, but it was too inflexible to be reused, so I decided to start over. To get decent performance out of A* you need a fast priority queue, so I figured that it would be a good first step.

You might want to download the sample projects if you want to follow along:

Test project for Xcode: CHeap
6502 code: AsmHeap.

Playing with priority queues

A priority queue is an abstract data type like a stack or a queue, where you can insert elements that have an associated priority, and its unique selling point is that you can quickly pull out the element with the highest priority. It’s commonly implemented using a heap. Python comes with a built-in module called heapq which uses a standard list to store the heap’s binary tree, so we can take a look at how it behaves:

$ python
Python 2.7.5 (default, Sep 12 2013, 21:33:34) 
[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import heapq
>>> tree = []
>>> for value in (6, 4, 7, 3, 8):
...     heapq.heappush(tree, value)
...                                                             3
>>> tree                                                       / \    Each child is
[3, 4, 7, 6, 8]                             Tree structure:   4   7   smaller than
>>> while tree:                                              / \      its parent.
>>>     print heapq.heappop(tree)                           6   8
... 
3
4
6
7
8

A nice side effect is that when you pull out each item in priority order you end up with sorted data, since a heap is at the base of heapsort. heapq implements a min-heap, so you get elements in ascending order.

High level implementation

Instead of hitting the metal right away I like to start with a high level implementation. Since I needed an excuse to play with unit testing in Xcode, Objective-C seemed like a good idea. An NSMutableArray is a comfortable choice for storing the tree, so let’s make a simple class interface:

@interface CASMutableArrayHeap : NSObject

@property NSMutableArray *tree;

- (int)size;
- (void)push:(NSNumber *)value;
- (NSNumber *)pop;

@end

You can find the class implementation in CASMutableArrayHeap.m on GitHub, since it’s a bit too big to paste here. It’s is based on chapter 10 in Mastering Algorithms with C by Kyle Loudon (O’Reilly 1999) who explains the algorithm much better than I ever could.

CAstarMutableArrayHeapTests.m contains unit tests that confirms that the classes grows and shrinks as expected, and that it correctly sorts a list of random numbers.

Low level C implementation

The next step on the way down is to get rid of the high level types and automatic memory handling, so goodbye NSMutableArray. Instead let’s work with a fixed buffer. This loses a lot of flexibility but the speed gains are massive:

**1000 * 128 push/pop**
Heap Implementation	Time
NSMutableArray	1431 ms
Fixed memory buffer	28 ms

This implementation can be found in CASCArrayHeap.m.

Pseudo-assembly

This is where I’d normally start writing assembly code, but since I’m having so much fun with the unit tests I figured I might as well write some 6502 pseudo code in C. This means that while it’s still valid C code, all data is now manipulated 8 bits at a time, all arithmetic is done with a single variable called a and all indexed addressing is done with x and y, mimicking the 6502 CPU registers. Readability has gone out the window, but it can be translated line by line into assembly and it’s nice to be able to test it for correctness before diving in. This version is in CASCAsmHeap.m.

Finally the promised 6502

With a solid understanding of the implementation it’s time to bring it down to assembly level. I prefer working with the assembler from the cc65 compiler suite, so let’s create a public interface in asmheap.h allowing the code to be called from C:

extern uint8_t asmheap_size;

void asmheap_init();
void __fastcall__ asmheap_push(uint8_t value);      /* Pass value argument in A/X instead of on the stack. */
uint8_t asmheap_pop(void);

The implementation starts with allocating a buffer for the tree:

	.align 128

tree:	.res 128	; Reserve 128 bytes for the tree, which allows
size:	.res 1		; us to use the N flag to check if it's full.

The first function is asmheap_init, which couldn’t be simpler:

; Initialize an empty heap.
_asmheap_init:
	lda #0
	sta size
	rts

For push and pop let’s use a small macro to help swap nodes in the tree:

; Swap node at x with node at y.
  .macro swap
	lda tree,x
	sta @temp
	lda tree,y
	sta tree,x
@temp = * + 1
	lda #$5e		; Self mod.
	sta tree,y
  .endmacro

Push:

; Push a value into the tree.
_asmheap_push:
	ldx size
	; If the heap is full, overwrite the last node.
	bpl :+
	dex
	stx size
:	; Push A to the bottom of the heap.
	sta tree,x
	inc size

	; Set Y to the parent node = (X - 1) / 2.
	txa
	sec
	sbc #1
	lsr
	tay
@heapify:
	; If X is at the top of the heap, we're done.
	cpx #0
	beq @done
	; If X is larger than its parent Y, we're done.
	lda tree,x
	cmp tree,y
	bcs @done
	; Swap X with its parent Y.
	swap
	; Set X to point at its parent.
	tya
	tax
	; Set Y to the new parent node.
	sec
	sbc #1
	lsr
	tay
	jmp @heapify
@done:
	rts

And pop:

; Pop the lowest value from the tree.
_asmheap_pop:
	; Save the item at the top so we can return it.
	lda tree
	sta @return_value
	; Remove the item at the bottom of the tree.
	dec size
	beq @done
	ldy size
	lda tree,y
	; Move it to the top of the tree.
	sta tree
	; Heapify the tree.
	ldx #0
	stx @current_node
@heapify:
	; Left child is at x * 2 + 1.
	txa
	asl
	;clc		; X is always < 128
	adc #1
	tay
	; Check if we're at the bottom of the tree.
	cpy size
	bcs @done
	; Check if left child is smaller.
	lda tree,y
	cmp tree,x
	; If not skip to checking the right child.
	bcs @check_right
	; Since it was smaller let X point to the left child.
	tya
	tax
@check_right:
	; Right child is at x * 2 + 2, or left + 1.
	iny
	; Check if we're at the bottom of the tree.
	cpy size
	bcs @check_swap
	; Check if right child is smaller.
	lda tree,y
	cmp tree,x
	bcs @check_swap
	tya
	tax

@check_swap:
	; If either child was smaller X is different from the current node.
@current_node = * + 1
	cpx #$5e		; Self mod.
	; If not we're done.
	beq @done
	
	; Swap parent with child.
	ldy @current_node
	swap
	; Let the child be the new current node.
	stx @current_node
	; Repeat.
	jmp @heapify
@done:
	ldx #0
@return_value = * + 1
	lda #$5e		; Self mod.
	rts

Testing and benchmarking

In order to verify that the final implementation works as intended I ended up writing a small test and benchmarking harness. You can find the heap implementation together with the tests on GitHub as AsmHeap. For fun let’s run the benchmark and see how it fares:

That’s about 86 seconds for 1000 * 128, which makes my 3.7 GHz Xeon E5 Mac Pro about 3000 times as fast as my C64.

AutoDMG v1.4

2014-04-15T00:00:00+00:00

AutoDMG 1.4 is here, and with its release I consider it feature complete: it now supports every feature that InstaDMG has. The last missing piece of the puzzle was BaseOS caching, but instead of just copying the feature I have a slightly more generalized take on it. You can now use a previously generated image as the source, allowing you to update images as needed or cache them at any point. You’ll find the details in the wiki.

Changes in 1.4:

Support for using previously generated images as sources.
Added Advanced Options window for setting the volume’s name and size.
Drag & drop apps can be added to additional software, and will be copied to /Applications.
Apps and packages can be wrapped in a dmg.
You can now load and save of templates from the GUI.
The template used to build the image is saved to /var/log inside the image for future reference.
Improved error handling should allow saving of the log if the app crashes.

Cached image as source, advanced options, drag & drop app, pkg in dmg, and new crash handler

You can download 1.4 from the AutoDMG releases page. If you have any questions join us in ##osx-server (but please check the documentation and especially the FAQ first).

AutoDMG v1.3.2

2014-03-27T00:00:00+00:00

uclara discovered a regression when building on 10.8.5.

You can download 1.3.2 from the AutoDMG releases page. If you have any questions join us in ##osx-server (but please check the documentation and especially the FAQ first).

AutoDMG v1.3.1

2014-03-26T00:00:00+00:00

Here’s a quick release of 1.3.1 since a major feature didn’t make it in time for 1.3: thanks to the work of FireAllianceNX 10.7 support has been re-enabled, and there’s an update profile available for 11G63.

And as there was a lot of code added in 1.3, it also came with its fair share of bugs. Thanks to Tim Sutton, Anthony Reimer, and dhandy0 the following have been found and squashed:

Fixed a crash if building in CLI without UpdateProfiles.plist in Application Support.
Fixed handling of relative paths in the CLI.
Fixed build failing if the output path pointed to a nonexistent directory.
Fixed building with packages stored on network volumes.
Fixed inaccurate error message when a broken package was added.

You can download 1.3.1 from the AutoDMG releases page. If you have any questions join us in ##osx-server (but please check the documentation and especially the FAQ first).

AutoDMG v1.3

2014-03-24T00:00:00+00:00

Following a few betas to test the waters AutoDMG 1.3 is finally ready for release. For those of you that like building with the GUI the only new feature is that the 32 GB limit is gone as AutoDMG will now automatically create a larger image if you add large packages. The biggest news is for everyone who likes automation and reproducible workflows, since AutoDMG finally has a proper command-line interface. Not that everyone has been sitting idly by, we’ve recently seen the release of some nice solutions from the folks at Google and Adobe, and an interesting way of merging InstaDMG’s workflow with AutoDMG. My hope is that 1.3 will integrate much smoother in workflows that require building large images, or when you need to build a lot of images repeatedly. The documentation on how to throw away the training wheels is in the wiki:

You can download 1.3 from the AutoDMG releases page. If you have any questions join us in ##osx-server (but please check the documentation and especially the FAQ first).

It seems appropriate to end with a screenshot:

$ cat 10.9.2.adtmpl
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>TemplateFormat</key>
    <string>1.0</string>
    <key>SourcePath</key>
    <string>~/Desktop/Install OS X Mavericks.app</string>
    <key>OutputPath</key>
    <string>~/Desktop/BaseImage-10.9.2-13C64.dmg</string>
    <key>VolumeName</key>
    <string>Macintosh HD</string>
    <key>ApplyUpdates</key>
    <true/>
    <key>AdditionalPackages</key>
    <array>
    </array>
    <key>IncludeTemplates</key>
    <array>
    </array>
</dict>
</plist>
$ /Applications/AutoDMG.app/Contents/MacOS/AutoDMG -l autodmg-10.9.2-20140324.log -L 7 build 10.9.2.adtmpl
Installer: /Users/admin/Desktop/Install OS X Mavericks.app
Output Path: /Users/admin/Desktop/BaseImage-10.9.2-13C64.dmg
Volume Name: Macintosh HD
Accepted source /Users/admin/Desktop/Install OS X Mavericks.app: Mac OS X 10.9.2 13C64
Found installer: Mac OS X 10.9.2 13C64
Update: iBooks Update (14.6 MB)
Update: BookKit Update (78.5 kB)
Update: iTunes 11.1.5 (232.7 MB)
Password for admin: ********
Starting build
Using installer: Mac OS X 10.9.2 13C64
Using output path: /Users/admin/Desktop/BaseImage-10.9.2-13C64.dmg
Starting task with 1 phases
Starting phase: Preparing
phase: Preparing
Starting task with 7 phases
Starting phase: Starting install
phase: Starting install
Install task running
Script phase: sparseimage
Starting phase: Creating disk image
phase: Creating disk image
Script phase: install 0:/private/tmp/dmg.vXAs2G/Packages/OSInstall.mpkg
Starting phase: Installing OS
phase: Installing OS
Script phase: install 1:/Users/admin/Library/Application Support/AutoDMG/Updates/iBooksDelta.pkg
Starting phase: Installing iBooks Update
phase: Installing iBooks Update
Script phase: install 2:/Users/admin/Library/Application Support/AutoDMG/Updates/BookKitDelta.pkg
Starting phase: Installing BookKit Update
phase: Installing BookKit Update
Script phase: install 3:/private/tmp/dmg.XQKgax/Install iTunes.pkg
Starting phase: Installing iTunes 11.1.5
phase: Installing iTunes 11.1.5
Script phase: asr
Starting phase: Finalizing disk image
phase: Finalizing disk image
Starting task with 4 phases
Starting phase: Finalizing disk image
phase: Finalizing disk image
Finalize task running
Script phase: asr1
Starting phase: Finalizing disk image
phase: Finalizing disk image
Script phase: asr2
Starting phase: Finalizing disk image
phase: Finalizing disk image
Script phase: asr3
Starting phase: Finalizing disk image
phase: Finalizing disk image
Starting task with 1 phases
Starting phase: Finishing
phase: Finishing
Finish
Build finished successfully, image saved to /Users/admin/Desktop/BaseImage-10.9.2-13C64.dmg
Build successful

Managing Java Security Site Exceptions With Deployment Rule Sets

2014-02-11T00:00:00+00:00

Like many organization we have a few legacy systems that require Java to run, and as much as we’d like to replace them we’re not quite there yet. With 7u51 Oracle turned up the default security settings by another notch (undoubtedly a good thing!) and as a result it will no longer allow users to run unsigned applets without a security manifest. Users that have admin privileges can whitelist applets themselves, and there are some creative ways of managing site exceptions, but the officially supported way is to create a Deployment Rule Set. It’s not particularly complicated but if you’re not a Java developer it involves some tools that you’re probably not familiar with (at least I wasn’t). Here’s my annotated guide to creating, signing, and deploying DeploymentRuleSet.jar.

1. Get a jar signing certificate

First off we’ll create a new keystore, and generate a self signed certificate and keypair:

$ keytool -genkey -alias MYORG -keyalg RSA -validity 1095 -keystore ~/MYORG_keystore

keytool will ask you to set a password for the keystore, and ask who the certificate is for. The certificate authority we use (TERENA Certificate Service) didn’t accept DSA keys so I had to set the algorithm to RSA. I also selected a 3 year validity period, in the vain hope that we’ll have replaced those old legacy systems by then (hello to myself googling this again in 3 years).

At this point you have two options:

Buy a code signing certificate from one of the CAs that are trusted by Java. Expect to pay at least $80/year.
Make your clients trust a self-signed certficiate. Free, involves a little more work.

If you get a certificate signed by a trusted authority, create a CSR that we can send off to a certificate authority:

$ keytool -certreq -alias MYORG -keystore ~/MYORG_keystore
Enter keystore password: 
-----BEGIN NEW CERTIFICATE REQUEST-----
MIIC+DCCAeACAQAwgYIxCzAJBgNVBAYTAlNFMRUwEwYDVQQIEwxZT1VSIENPVU5UUlkxEjAQBgNV
BAcTCVlPVVIgQ0lUWTEaMBgGA1UEChMRWU9VUiBPUkdBTklaQVRJT04xGDAWBgNVBAsTD1lPVVIg
REVQQVJUTUVOVDESMBAGA1UEAxMJWU9VUiBOQU1FMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
CgKCAQEAqx4D8UoKecvVcSDkJWkLBr94IOb+aZuW704dfteg/scNHtsklAza/Taj/qcQLUnH+/BC
QW4HeOibtKMnvtDoz9qo2Cx0O+qJ4+XYXO2wGAorh1nvqbfmFCy/nmgRTJcUgROCW99FI0knlB2Z
fkeXSKQuu2wnLadflra/RmFBVcECSigBcQVG4xUBpRjxxIsM84hh0bez1KZgjRW66o62AbFynIsn
NsMB11d+1rUmZFVBqFwNcZ+CiKJfpOrBgLxQayRed/Jxhv+IKT95vz7a7czvfTzoopr9OuTqK/rC
d3V04744kFFjIUhNahbU4rOZtkr4htMAF17HFe61OaxbUQIDAQABoDAwLgYJKoZIhvcNAQkOMSEw
HzAdBgNVHQ4EFgQUgQjjFrERujTjUjlSIl6ia6l47ykwDQYJKoZIhvcNAQELBQADggEBACSH03D+
QSDbMp350+nUVgiV1tseInJJLkTfEG3F9lZPnJcdeY4rpb/JUCubmqD58iM5gHojNsrIucyY2Opp
PDE5lqD0oth3e48gkTL4VDuMLAk4tyGFVTuWHTGASf3b446qX5eufTJD3ri6bk7mJ8qXt8eDm4uB
z5+aNZ4m/Oh/pTOZxcnfXWGmO7yxIGfUDkiROvQm46qZ5pFwE+THNcPqIWCPLoiaEfdVLbQS6/ie
KNbVqnG0TtnCnhCnTIdWs2JXpniuMt9bpZs5n2nsOQYq1BxstLh8uQx6VhnFhc2h8lJbaXjC+miD
gPfUMYEiUNun30VaYBtfEJQJQlpBbtI=
-----END NEW CERTIFICATE REQUEST-----

Paste this into the CA’s request form, retrieve the signed certificate, and replace the self-signed one:

$ keytool -importcert -file ~/Downloads/signed_cert.pem -trustcacerts -alias MYORG -keystore ~/MYORG_keystore

You should now have a valid code signing certificate in your keystore, and you’re ready to create and sign jar files.

2. Create DeploymentRuleSet.jar

Oracle gives you a fair bit of control over how applets should run (refer to the Deployment Rule Set documentation here), but here’s an example of a very simple whitelist for two sites, while leaving everything else to run with the default security policy:

<?xml version="1.0" encoding="UTF-8"?>
<ruleset version="1.0+">
    <rule>
        <id location="https://cruft.example.com" />
        <action permission="run" />
    </rule>
    <rule>
        <id location="https://dinosaur.example.com" />
        <action permission="run" />
    </rule>
    <rule>
        <id /><!-- The last rule is the default policy and the id should be blank. -->
        <action permission="default" />
    </rule>
</ruleset>

Then we package ruleset.xml into DeploymentRuleSet.jar:

$ jar cf DeploymentRuleSet.jar ruleset.xml

and finally sign DeploymentRuleSet.jar with the certificate:

$ jarsigner -tsa https://timestamp.geotrust.com/tsa DeploymentRuleSet.jar MYORG

3. Distribute DeploymentRuleSet.jar to your clients

With the jar packaged and signed, all that’s left to do is distribute it to your clients. No fancy magic is needed here, just drop it in the appropriate path:

OS	Path
Windows	`C:\Windows\Sun\Java\Deployment\DeploymentRuleSet.jar`
Mac OS X	`/Library/Application Support/Oracle/Java/Deployment/DeploymentRuleSet.jar`
Linux & Solaris	`/etc/.java/deployment/DeploymentRuleSet.jar`

4. Add a reminder for when the certificate expires

Remember that certificates expire, and you really don’t want to be caught with your pants down. I usually create a calendar reminder 3 months before it expires, with all the details needed to renew it.

References

The joys of an Intel X25-M SSD in a MacBook Pro 15" (Early 2011)

2014-02-05T00:00:00+00:00

I got my shiny new Mac Pro the other week, and I finally finished migrating everything from my old machine. A nice side effect was that I could move the SSD over from my old Mac Pro to my laptop, since it’s the last of my machines suffering from spinning rust. Heck, even my C64s and Amigas have flash drives these days. The early 2011 15" is easy enough to open up, and Thunderbolt target mode makes it easy to shuffle the data over.

Unfortunately the machine refused to boot afterwards, and in recovery mode the SSD wasn’t even visible in Disk Utility or system_profiler, yet it showed up as a device in the Startup Manager and obviously worked in target mode. There are lots of threads on the discussion boards about trouble with SSDs in the 2011 MacBooks, apparently due to badly shielded SATA cables. It turned out to be a red herring though, as the actual problem turned out to be a firmware bug:

(2CV102M3) This firmware revision fixes enumeration and slow boot issues […]

As soon as Spotlight and Dropbox stops fighting over who gets to index the drive first my laptop should finally join the ranks of snappy computers.

And special thanks to Cpuroast in ##osx-server on helping me find the intel fw update.

AutoDMG v1.2.1

2013-12-17T00:00:00+00:00

I’ve been working on implementing caching in AutoDMG, but the bugs kept piling up in the meantime. I decided to break the silence with a small bugfix release:

Allow renamed InstallESD images (but they still have to match InstallESD*.dmg).
Fix message regarding deprecated installers on 10.8.
Fix a crash if the user’s primary GID doesn’t have a corresponding group.
Fix deletion of multiple additional packages.

I also published new update profiles for 10.9.1 and 10.8.5 earlier today which should be downloaded automatically.

You can download 1.2.1 from the AutoDMG releases page. If you have any questions join us in ##osx-server (but please check the documentation and especially the FAQ first).

AutoDMG v1.2

2013-11-14T00:00:00+00:00

What a week.

Friday started with the new Retina MacBooks arriving, the ones that can’t be downgraded to 10.8. We have a working solution for upgrading machines to Mavericks but we’re still not ready with the deployment workflow for 10.9 since we’ve been spending the last few weeks chasing PHD sync bugs with NetApp. In the middle of getting the 13A2093 and 13A3017 builds ready my Mac Pro decided to keel over and die. It came back to life again, and I guess I should be only thankful for an excuse to get an Epic Thermos of Power.

Then on Monday DeployStudio flaked out and the Finalize task started using 100% CPU taking down package installs with it. Our Absolute Manage server also decided that it’d had enough and started telling the clients to go away since it needed its beauty sleep.

Tuesday started with a bit of relief as we got some help from Absolute tweaking the server settings, and it’s been behaving since. Profile Manager had been coughing and wheezing though and on Wednesday it crapped out totally. Today was spent migrating profiles out of Profile Manager and into Absolute Manage, something we’d been planning to do anyway as part of our move to 10.9, just not this week…

Despite all this AutoDMG version 1.2 is out, but only with a few minor changes. Firstly I now set CM_BUILD=CM_BUILD which makes installer bypass packages’ volume checks. This means that you can create machine specific builds (like 13A2093 and 13A3017) without having to run AutoDMG on that particular hardware. It also makes it possible to install packages that are configured to only install on root volumes. This should bring AutoDMG in line with InstaDMG, for better or for worse.

After being prodded by Tuomo Oksakivi I also took a look into why aliases didn’t always work with drag & drop. Turns out that when you’re working with paths you have to resolve aliases manually, but it was simple enough to fix. The idea here is that if you have a set of packages that you want to include in every image you build, you can create a folder with aliases for easy dragging & dropping. It’s not exactly a replacement for InstaDMG catalogs, but it might help you keep your builds a little more organized. I have an idea for image templates, but we’ll have to see if I have time to work on it.

But enough rambling. You can download 1.2 from the AutoDMG releases page. If you have any questions join us in ##osx-server (but please check the documentation and especially the FAQ first).

Now I think I deserve a proper cup of tea.

AutoDMG v1.1

2013-11-06T00:00:00+00:00

AutoDMG version 1.1 has been released, with the following changes:

Added support for InstallESD.dmg as source from Timothy Sutton.
Better logging during package installs.
Fix packages failing if they were in a folder with non-ascii characters in the name (thanks to Darkomen78).

You can download 1.1 from the AutoDMG releases page. If you have any questions join us in ##osx-server (but please check the documentation and especially the FAQ first).

If you report a bug, please save the log file and attach it to the issue, it’s almost impossible to diagnose problems without it.

Accessing Kerberized Services with the Active Directory Computer Account

2013-11-05T00:00:00+00:00

Back in May I posted an article on how to access kerberized services with the AD computer account. If the blog statistics are to be believed it’s a fairly popular post too. Well, it turns out that I’d basically just missed -k when I read the man page for kinit. I also managed to expose the machine’s password in the process of doing so, making it even more embarrassing.

I updated the original article, but for reference here’s a proper script:

#!/bin/bash
#
# This script requires 10.7+ and root privileges.

# Get Kerberos ticket for the machine.
declare -r ADPLIST=$(echo "/Library/Preferences/OpenDirectory/Configurations/Active Directory"/*.plist)
declare -r ADCOMPACCT=$(/usr/libexec/PlistBuddy -c 'print :trustaccount' "$ADPLIST")
kinit -k "$ADCOMPACCT"

# Access services with Kerberos credentials.
mount_smbfs //sharedfiles.server.com/SharedFolder /tmp/sharedfolder
curl --negotiate -u : http://server.example.com/protectedresource

I’ll go stand in the corner now.

AutoDMG v1.0

2013-11-01T00:00:00+00:00

I’ve received some great feedback during the past few days, but since no major issues have been reported I’m feeling confident enough to release AutoDMG version 1.0.

A big thank you goes out to everyone who helped with the testing.

The changes for 1.0 include:

Polished UI.
Better logging.
Fix list of updates being refreshed after profiles updated.
Added warnings for deprecated installers.
Confirm to quit when building.
Added preferences for automatic updates.

You can download 1.0 from the AutoDMG releases page. If you have any questions join us in ##osx-server (but please check the documentation and especially the FAQ first).

Create Recovery Partition Installer v1.1

2013-10-30T00:00:00+00:00

Timothy Sutton submitted a patch yesterday that brings 10.9 support to Create Recovery Partition Installer. Grab it on GitHub:

https://github.com/MagerValp/Create-Recovery-Partition-Installer/releases

AutoDMG

2013-10-29T00:00:00+00:00

Since InstaDMG has issues with 10.9 and SIC is no more I took it upon myself to create a replacement, and the result is AutoDMG. It uses the same mechanic as InstaDMG to generate never booted system images, and also applies the latest software updates and installs your own custom packages. The biggest change is that it’s a GUI application now:

Today I’m releasing 0.8 which has matured a lot over the previous releases, and it’s the first version I feel is ready for serious testing. There are most likely still bugs lurking around which is why I’m not calling it 1.0 just yet, but it’s working well here for building images of 10.8.5 and 10.9.

So, please grab it and test it in your workflows. If you have any questions please check out the documentation first (yes, I wrote some!) and join us in ##osx-server to discuss it.

I should also thank Duncan McCracken for showing us how to install OS X on a disk image at MacSysAdmin, which was the seed that grew into AutoDMG. Also thanks to the unstoppable Greg Neagle who came up with the name. Thanks guys!

PS: If anyone’s wondering what’s up with CreateUserPkg, there’s a beta of 1.2.4 hiding in the issue tracker that I’m going to package up and release as soon as AutoDMG settles down.

Irony is a B*tch

2013-09-13T00:00:00+00:00

So yesterday I posted about a vulnerability is OS X where passwords are leaked during login, and by chance I happened to read my previous post on accessing kerberized services with the Active Directory computer account today. It prominently featured the following nugget:

echo "$ADCOMPPWD" | kinit --password-file=STDIN "$ADCOMPACCT"

Oh sweet irony. I updated the post with a fixed version that uses a temporary file instead of piping from echo for the whole world to see.

OS X Passwords Leaked During Login

2013-09-12T00:00:00+00:00

NOTE: This post was written May 17th but was withheld until Apple patched the vulnerability.

Overview

When logging in on a machine running OS X 10.7 - 10.8.4 with a configuration profile applied passwords are exposed in clear text to other logged in users.

Apple has released 10.8.5 (combo) and Security Update 2013-004 for 10.7 with patches for the vulnerability.

CVE-ID

CVE-2013-1030 (Apple’s summary)

Background

I was writing a script today to automatically mount shared folders for our users when they log in. I wanted the script to wait for the desktop to appear before it runs, so users can see the status window in case there is a problem with the mount. The easiest way to accomplish this is to wait for a process to appear (e.g. Finder.app) and then wait a couple of seconds more for good measure. To find out what processes appear during a typical login I wrote the following snippet and ran it in the background while I logged in with my test user:

while true; do
    echo "----------------"
    ps auxww | grep [u]sername | cut -c70- | sort
    sleep 1
done

This confirmed my suspicion that the best option seems to be to wait for Finder.app and Dock.app to appear, but what nearly caused a [T N>K](http://www.catb.org/jargon/html/C/CNK.html) were the first snapshots of the process table:

----------------------------------
/System/Library/CoreServices/FileSyncAgent.app/Contents/MacOS/FileSyncAgent -launchedByLaunchd -PHDPlist
/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/CVMCompiler
/System/Library/PrivateFrameworks/SystemMigration.framework/Resources/Tools/migCacheCleanup -JustPhotoStreamNudge
/System/Library/PrivateFrameworks/TCC.framework/Resources/tccd
/sbin/launchd
/usr/libexec/xpcd
/usr/sbin/cfprefsd agent
/usr/sbin/distnoted agent
----------------------------------
/System/Library/CoreServices/FileSyncAgent.app/Contents/MacOS/FileSyncAgent -launchedByLaunchd -PHDPlist
/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/CVMCompiler
/System/Library/PrivateFrameworks/SystemMigration.framework/Resources/Tools/migCacheCleanup -JustPhotoStreamNudge
/System/Library/PrivateFrameworks/TCC.framework/Resources/tccd
/sbin/launchd
/usr/libexec/mdmclient mcx_userlogin testuser secretpwd
/usr/libexec/xpcd
/usr/sbin/cfprefsd agent
/usr/sbin/distnoted agent
----------------------------------
/System/Library/CoreServices/FileSyncAgent.app/Contents/MacOS/FileSyncAgent -launchedByLaunchd -PHDPlist
/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/CVMCompiler
/System/Library/PrivateFrameworks/SystemMigration.framework/Resources/Tools/migCacheCleanup -JustPhotoStreamNudge
/System/Library/PrivateFrameworks/TCC.framework/Resources/tccd
/sbin/launchd
/usr/libexec/xpcd
/usr/sbin/cfprefsd agent
/usr/sbin/distnoted agent

It’s fascinating how jarring it is to see your password in clear text, the reaction is almost physical. I wrote up a quick bug report (radar 13921614), contacted product-security@apple.com, and confirmed that it’s present all the way back to 10.7, but a mitigating circumstance is that the machine needs to have a configuration profile applied, e.g. by being enrolled with Profile Manager.

To test it, log into a machine with a configuration profile applied as a standard (unprivileged) user either over SSH or with fast user switching enabled, and execute:

$ while true; do ps auxww | grep '[m]dmclient mcx_userlogin'; done

Then log in with a different user account at the login window, and watch the shell’s output. It’s like nails on chalkboard.

If you’re managing student labs or other public machines I’d recommend applying the appropriate patches ASAP.

Accessing Kerberized Services with the Active Directory Computer Account

2013-05-22T00:00:00+00:00

The usual scenario when you’re in a kerberized environment is that your users get a Kerberos ticket on login and they use this to access protected services, such as their home directory. Sometimes you need to execute a task in the background, and not necessarily on behalf of a user or even with anyone logged in, but how do you guarantee that the connecting client is who it says it is? In these situations you can use the machine’s AD computer account for authentication:

#!/bin/bash
#
# This script requires 10.7+ and root privileges.

# Initialize Kerberos if needed.
#if ! klist -t 2>/dev/null; th ##########
#    # Read the AD configu ##################
#    declare -r ADPLIST= ######          ###### nces/OpenDirectory/Configurations/Active Directory"/*.plist)
#    declare -r ADCOMP ####     ALWAYS      ##### dy -c 'print :trustaccount' "$ADPLIST")
#    declare -r ADNO ####                 #### #### 'print :"node name"' "$ADPLIST")
#                   ####    READ THE    ####    ####
#    # Read the com ###               ####       ### stem keychain.
#    declare -r SK ####  MAN PAGE   ####         #### n"
#    declare -r TE ####           ####           ####
#    security find ####         ####   BEFORE    #### SKC" 2>&1 >/dev/null | cut -d\" -f2 > "$TEMPFILE"
#    # 10.8+ suppor ###       ####               ###
#    #security find ####    ####    POSTING     #### "$SKC" > "$TEMPFILE"
#                    #### ####                 ####
#    # Use the compute #####     !!!!!!      #### d get a Kerberos TGT.
#    kinit --password-fi ######          ###### "$TEMPFILE"
#    rm -f "$TEMPFILE"     ##################
#fi                            ##########

# Get Kerberos ticket for the machine.
declare -r ADPLIST=$(echo "/Library/Preferences/OpenDirectory/Configurations/Active Directory"/*.plist)
declare -r ADCOMPACCT=$(/usr/libexec/PlistBuddy -c 'print :trustaccount' "$ADPLIST")
kinit -k "$ADCOMPACCT"

# Access services with Kerberos credentials.
mount_smbfs //sharedfiles.server.com/SharedFolder /tmp/sharedfolder
curl --negotiate -u : http://server.example.com/protectedresource

Update: Fixed security find-generic-password which doesn’t accept -w on 10.7, thanks @RegalWeasel.

Update 2: Fixed ironic exposure of computer account password in process table.

Update 3: Read the man page and removed 14 lines of bash.

CreateUserPkg v1.2.3

2013-04-05T00:00:00+00:00

CreateUserPkg v1.2.3 is now available in the App Store and on the project homepage.

Making the Impossible Possible

2013-03-31T00:00:00+00:00

There’s an old joke in the Commodore 64 scene that if you want to see something done, just claim that it’s impossible and make sure Crossbow hears you say it. The C64 celebrated its 30th birthday last year, and the limits of this unassuming little brown box are still nowhere to be found.

Back in 2006 while I was working on Ultima IV Gold I’d established that none of the available loader systems met the requirements for the project, the biggest issue being that I needed to squeeze almost 800 files into a disk format that only has 683 physical sectors. So armed with Lasse Öörni’s rants and Inside Commodore DOS I sat down and eventually managed to create my own system, retronymed Uload. In the following year I also tackled Ultima III Gold, and with lots of help from Krill I experimented with fast GCR decoding and network transfer tools. So armed with knowledge and experience I inevitably performed the fatal mistake, documented on the internet for all eternity:

“1541 isn’t fast enough to decode in realtime.”

Well, shucks. While Crossbow wasn’t listening, it turns out that LFT was. He released Shards of Fancy today, and the accompanying article on GCR decoding on the fly is now up. Hats off to you, Linus.

Bonus anectode: Having misinterpreted the JPEG spec back in the 90s I argued quite vehemently that you couldn’t decode it on a C64, due to a lack of RAM. The result of course was Juddpeg.

Poking Around in Mac App Store Receipts

2013-03-19T00:00:00+00:00

Yngve Åström asked if anyone knew how to tell which Apple ID installed an app store app on the MacEnterprise mailing list:

Is it just me or have anyone been able to read something useful out of the app/Contents/_MASReceipt/receipt? I’m looking for a way to find out which account was used to by an app, the _MASreceipt library looked like the right place to look. Doesn’t matter how I trie to read the receipt all I get is bits of readable info that makes very little sense to me. Looks like binary peaces of certs but nothing close to an AppStore account. The /Users/myuser/Library/Preferences/com.apple.storeagent.plist AppleID will tell me what account I’m using now but that’s it. Where can I find which account was used to by a particular app? In this case Server.app… Is it even possible to find out?

Let’s poke around and see what’s inside those receipt files the Mac App Store puts inside every app bundle. The receipt file itself is a PKCS #7 container, as defined by RFC2315, with its payload encoded using ASN.1. We can look at its certificates using openssl’s pkcs7 command:

$ openssl pkcs7 -inform der -in /Applications/Xcode.app/Contents/_MASReceipt/receipt -print_certs -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            18:59:43:21:72:74:9c:fc
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=US, O=Apple Inc., OU=Apple Worldwide Developer Relations, CN=Apple Worldwide Developer Relations Certification Authority
        Validity
            Not Before: Nov 11 21:58:01 2010 GMT
            Not After : Nov 11 21:58:01 2015 GMT
        Subject: CN=Mac App Store Receipt Signing, OU=Apple Worldwide Developer Relations, O=Apple Inc., C=US
[...]

You’ll see that it contains a certificate chain with Apple’s root CA, the developer CA, and the receipt signing certificate. There’s a signed payload in the receipt as well, which we can see if we dump the ASN.1 data using openssl’s asn1parse command:

$ openssl asn1parse -inform der -in /Applications/Xcode.app/Contents/_MASReceipt/receipt
    0:d=0  hl=4 l=4701 cons: SEQUENCE          
    4:d=1  hl=2 l=   9 prim: OBJECT            :pkcs7-signedData
   15:d=1  hl=4 l=4686 cons: cont [ 0 ]        
   19:d=2  hl=4 l=4682 cons: SEQUENCE          
   23:d=3  hl=2 l=   1 prim: INTEGER           :01
   26:d=3  hl=2 l=  11 cons: SET               
   28:d=4  hl=2 l=   9 cons: SEQUENCE          
   30:d=5  hl=2 l=   5 prim: OBJECT            :sha1
   37:d=5  hl=2 l=   0 prim: NULL              
   39:d=3  hl=4 l= 526 cons: SEQUENCE          
   43:d=4  hl=2 l=   9 prim: OBJECT            :pkcs7-data
   54:d=4  hl=4 l= 511 cons: cont [ 0 ]        
   58:d=5  hl=4 l= 507 prim: OCTET STRING      [HEX DUMP]:318201F7300A0201120201↩
0104021600300A02011302010104020C00300B02010E0201010403020101300C02010A0201010404↩
1602342B300C02010B0201010404020207D3300C02010D0201010404020227D9300E020101020101↩
040602041DABD29B300E020104020101040602043D83F593300E0201090201010406020450323132↩
300E0201100201010406020400DE7D2B300F02010302010104070C05342E362E31301002010F0201↩
01040802063C079CDD8EF6301B02010002010104130C1150726F64756374696F6E52656365697074↩
301C02010202010104140C12636F6D2E6170706C652E64742E58636F6465301C0201050201010414↩
80594D99A2FC354866BD2E624356CAA402371E8B301E02010802010104161614323031332D30332D↩
31355430363A31373A32365A301E02010C02010104161614323031332D30332D31355430363A3137↩
3A32365A3047020107020101043FA2F2AB232A8EE238211B3976878A67518DE97316756762D19CBD↩
6BAF021269B097CB4557693759139ECBDCC0526F788B2FB055E68B60B3555320FA64AE5905306102↩
01060201010459706C1F223E8C5CA8B556E292C13C6A1669C0BF7BC47B7B268C017B2D37F6BEA7BF↩
D4B4FFF5FF56DBEC269CCA6EEABFC6BC24C0185FFD7F9538088B0E6A99DA608B3129C81A64966151↩
E4C5B3931ECBF73EE7040949A1F46884
  569:d=3  hl=4 l=3669 cons: cont [ 0 ]        
  573:d=4  hl=4 l=1387 cons: SEQUENCE          
  577:d=5  hl=4 l=1107 cons: SEQUENCE          
  581:d=6  hl=2 l=   3 cons: cont [ 0 ]        
  583:d=7  hl=2 l=   1 prim: INTEGER           :02
  586:d=6  hl=2 l=   8 prim: INTEGER           :1859432172749CFC
  596:d=6  hl=2 l=  13 cons: SEQUENCE          
  598:d=7  hl=2 l=   9 prim: OBJECT            :sha1WithRSAEncryption
[...]

The first big blob of hex is a payload signed with sha1. This payload is also encoded using ASN.1, so let’s decode it and save it as a separate file, payload.asn:

$ openssl asn1parse -inform der -in /Applications/Xcode.app/Contents/_MASReceipt/receipt | grep -m 1 'OCTET STRING' | cut -d: -f4 | xxd -r -p > payload.asn

With it saved to disk let’s see what asn1parse has to say about it:

$ openssl asn1parse -inform der -in payload.asn
    0:d=0  hl=4 l= 503 cons: SET               
    4:d=1  hl=2 l=  10 cons: SEQUENCE          
    6:d=2  hl=2 l=   1 prim: INTEGER           :12
    9:d=2  hl=2 l=   1 prim: INTEGER           :01
   12:d=2  hl=2 l=   2 prim: OCTET STRING      [HEX DUMP]:1600
   16:d=1  hl=2 l=  10 cons: SEQUENCE          
   18:d=2  hl=2 l=   1 prim: INTEGER           :13
   21:d=2  hl=2 l=   1 prim: INTEGER           :01
   24:d=2  hl=2 l=   2 prim: OCTET STRING      [HEX DUMP]:0C00
   28:d=1  hl=2 l=  11 cons: SEQUENCE          
   30:d=2  hl=2 l=   1 prim: INTEGER           :0E
   33:d=2  hl=2 l=   1 prim: INTEGER           :01
   36:d=2  hl=2 l=   3 prim: OCTET STRING      [HEX DUMP]:020101
[...]

We can see that the payload is composed of a set of attributes, defined by two integers and an octet string. The first integer is the attribute type, the second its version (so far always 1), and the octet string its value. How the octet string is interpreted depends on the attribute type, and Apple has reserved most for private use, but a few are public:

Type	Definition	Value Interpretation
2	Bundle identifier	UTF8STRING.
3	Application version	UTF8STRING.
4	Opaque value	A series of bytes.
5	SHA-1 hash	20-byte SHA-1 digest value.
17	In-app purchase receipt	Further down the ASN.1 rabbit hole.

Bundle ID and app version are self explanatory, and the SHA-1 hash is used to verify that the app bundle hasn’t been modified. IAPs have their own receipt following the same principle as the main receipt, leaving only the mysteriously named “Opaque value”:

[...]
  114:d=1  hl=2 l=  14 cons: SEQUENCE          
  116:d=2  hl=2 l=   1 prim: INTEGER           :04
  119:d=2  hl=2 l=   1 prim: INTEGER           :01
  122:d=2  hl=2 l=   6 prim: OCTET STRING      [HEX DUMP]:02043D840EE2
[...]

From Apple’s documentation we can see that it’s used together with the computer’s GUID in computing the verification hash. The ASN.1 specification tells us that it’s an integer (0x02) and that it’s four bytes long (0x04). Let’s print it as a decimal integer:

$ printf "%d\n" 0x$(openssl asn1parse -inform der -in payload.asn | grep -A 2 ':04$' | tail -1 | cut -d: -f4 | cut -c5-)
1032064738

Still pretty opaque. Haven’t I seen that integer somewhere else though? Indeed I have:

$ defaults read com.apple.storeagent
{
    AccountKind = 0;
    AccountURLBagType = production;
    AppleID = "magervalp@mac.com";
    CreditDisplayString = "";
    DSPersonID = 1032064738;
    DownloadLocation = "/Users/pelle/Library/Application Support/AppStore";
    EligibilityCheckDate = "2013-03-18 22:26:53 +0000";
    ISBadgeValues =     {
    };
    ISLastUpdatesQueueCheck = "2013-03-15 06:17:25 +0000";
    KnownAccounts =     (
                {
            AccountKind = 0;
            AccountURLBagType = production;
            CreditDisplayString = "";
            DSPersonID = 1032064738;
        }
    );
    LastAuthTime = "2013-03-14 21:18:37 +0000";
    LastSynchedStoreFront = "143456,12";
    PurchasesInflight = 0;
    Storefront = "143456-17,13";
    UserNotificationDate = "2013-03-15 06:45:07 +0000";
}

Bingo - it’s the numeric app store user ID tied to my Apple ID. While we can’t directly determine which Apple ID installed a certain app, we can build a list of DSPersonIDs and their corresponding Apple IDs and get it that way.