Recent Books

In the order of reading

House of Suns – Alistair Reynolds

4/5. Space aristocrats uncover a terrible secret about their past in this galactic epic spanning millions of years. I had trouble putting it down and enjoyed the characters, plot and the every-technology-but-FTL setting. Thematically the lack of agency for the quadrillions of non-space-billionaires in the story is a little disappointing but not unexpected. There are plot holes, which are admittedly probably impossible to avoid in a story of this scope.

A Master of Djinn – P. Djeli Clark

4/5. An investigator unravels a sinister plot in a WW1 era Cairo Egypt full of of djinn, clockwork, angels, and magic. Enjoyed this book with reservations. Loved the costume and visual descriptions, especially learning many new words (Kaftan? Jambiya?). The setting is novel and interesting but could have been a little more internally consistent and solid.

A Memory Called Empire / A Desolation Called Peace – Arkady Martine

5/5. An ambassador to a stellar empire navigates between her love for the imperial culture and loyalty to her native station state. Loved these books. Many science fiction authors nail the technology but appear utterly clueless when it comes to the mechanisms of culture (KSR, looking at you). Not here. The interpersonal interactions are sharp and intelligent, and I loved the characters and the Aztec/flower/blood sacrifice world building. The aliens are alien. The people are alien. The languages are linguistic. It’s great.

Consider Phlebas – Ian M Banks

5/5. I have never read a not-great book by Ian M Banks, may he RIP. This is the first Culture novel. I admire greatly the way he imagines a flawed but idealistic civilization, dealing with the chaos and cruelty of reality. Flawed people doing their best. Just a great story.

Randomize – Andy Weir

3/5. A quantum casino heist goes awry. Decent story but not super memorable.

Emergency Skin – N.K. Jemsin

3/5. Clueless drone from a libertarian billionaire civilization returns to a solarpunk earth. I really loved the Stone Sky books, and my politics almost totally align with the author but this one was a bit too much of a political polemic for me. I want to see a realistic, flawed, yet optimistic vision of the future, but this wasn’t it.

Jonathan Strange & Mr. Norrell – Susana Clarke

5/5. Historical fantasy following two magicians as Fae magic returns to 19th century England. Writing style is reminiscent of Jane Austen and Oscar Wilde, but thematically modern. Great characterization of different types of people. A little slow in parts, but feels like working through a magnum opus. I couldn’t help buying some tarot cards after reading.

Elder Race – Adrian Tchaikovsky

4/5. A depressed but highly technological anthropologist helps save a lost colony fallen to medieval tech. Setting is reminiscent of “A Deepness In the Sky” by Vernor Vinge. Alternates between the medieval princess’ interpretation in high fantasy prose and the sarcastic technocratic but very depressed protagonist’s recollections. A good story, interesting setting, and I always enjoy looking at the same events from several perspectives.

Dying Earth – Jack Vance

4/5. A rogue adventures on an earth full of fantastic creatures and devices millions of year in the future. Enjoyed this book overall but felt frustrated with the protagonist Cudgel at points. A definite classic, the magic system in this book inspired D&D. I really enjoyed the super far-future fantasy setting and all the world building surrounding it.

Hail Mary – Andy Weir

5/5. Funny astronaut uses basic science to save humanity, with a serendipitous friend. Couldn’t put it down and screwed up my sleep schedule for the week. Andy Weir’s “Artemis” was perhaps more relevant for Extrapolation but this book plays to his strengths better. I admire the way he cleverly designed the premise to allow interstellar travel with current technology.

Network Effect – Martha Wells

3/5. Continuing adventures of an escaped corporate cyborg. I really loved the earlier Murderbot books, but I felt like they were running out of steam here.

The last Emperox – John Scalzi

4/5. Unlikely space empress outmaneuvers political opponents. I enjoyed reading this series. Solid writing craft. The mechanics of trade in the Interdependency were particularly interesting because I had been thinking about how trade should work in Extrapolation. The non-linear connection between Interdependency settlements are not unlike the actual deltav relationships between bodies in our solar system.

Announcing Extrapolation

Anisoptera Games Extrapolation

SPACE SETTLEMENT BUILDING AND MANAGEMENT GAME

I’d like to announce Anisoptera Games’ Extrapolation, a space settlement building and management game set in a realistic model of our solar system. You start out launching robotic probes from earth, set up little bases on the Moon and Mars, and gradually build up to space cities, giant rotating space habitats, orbital mirrors, space elevators, Dyson sphere, etc.

EXPLORE SETTLEMENT STRATEGIES

People discuss different approaches to space settlement. Should we build a Moon base first or go directly to Mars? What about mining the asteroids to build a big space station near Earth? How much would electromagnetic mass drivers or nuclear rockets help compared to conventional rockets? What kind of mining and launch operation would support a billion people in space? These are the kind questions I want to explore with this game.

SUPPLY MANAGEMENT

Extrapolation is about building a supply chain in space. You decide where to build satellites, space stations, factories, mining robots, launch sites, etc, and the existing units will try and move resources around to make that happen. It’s a cross between fortress management games like Oxygen Not Included or Dwarf Fortress and a solar system simulator like Celestia.

REALISTIC SOLAR SYSTEM

The Extrapolation model of our solar system uses data from actual space probes and telescopes. Rockets use realistic amounts of energy and propellant and follow elliptical trajectories between planets. Build bases in existing craters on the moon and explore thousands of asteroids. Bring water and ammonia in from the outer solar system, or search for ice in the perpetual darkness of the lunar south pole. Speed up time as you wait years for probes to arrive at Saturn.

PRE-ALPHA

Extrapolation is currently in pre-alpha. Most of the major systems have been implemented, and it’s sort of playable, but it needs a lot of work. It’s time to start talking about the game and incorporating feedback. I need help refining the interface and deciding which areas to focus on (Interstellar starships? Terraforming? Beam-rider ships?)

SIGN UP FOR TESTING OR UPDATES

Add your email address to the form below to get an email when there are Extrapolation updates. Fill out the survey to sign up for the alpha test. Join the discord chat with other people about Extrapolation (and related topics). Send me an email! Leave a comment! Tell your friends!

ALPHA TEST SIGNUP SURVEY

Extrapolation Discord channel

Placing a station in orbit around Earth
Sending resources (and people) from Earth to Mars, and beaming data back

Masses of Space Habitats

I did some calculations on popular cylindrical rotating space habitat designs, from the almost-practical Kaplana One to the ridiculously huge Banks Orbital. What would they be made from, and how much would they weigh? The O’Neill Island Three Cylinder was designed to the theoretical limit of steel, and the Mckendree Cylinder to the theoretical limit of carbon nanotubes, but are those really the best materials? And how much would we need? How much material could we save by reducing the gravity or air pressure? What kind of material would we need to construct a Banks Orbital?

NameLength (km)Radius (km)Floor Area (km2)EarthsCap height (km)People (dense)People (sparse)People (really sparse)
Kaplana One0.3250.250.51<<1%0.253,00226016
O’Neill Cylinder324804.250.00016%44,730,869410,16625,736
McKendree Cylinder46004601.33E+072.60%46078.2e+96.8e+9425.4e+6
Bishop Ring50010003.14E+060.61%20018.5e+91.6e+9100.5e+6
Banks Orbital30001.50E+062.83E+105531.13%200166.3e+1214.4e+12904.8e+9
The dense population figure is 170 m2/person, from the Kaplana One paper.
The sparse figure is 510 people/km2, the density of South Korea.
Really Sparse is 32 people/km2, the density of the United States

Structurally, space habitats are a cross between a balloon or pipe and a flywheel. Much of the pressure that the structure needs to contain comes from air, with the mass needed for radiation shielding also a significant consideration. Air pressure on earth is 15 psi, or 10 tonnes per square meter. That’s five 2 ton cars per square meter! Smaller habitats would be completely full of air at an essentially uniform pressure, while McKendree Cylinder sized structures would have a vacuum in the center (on Earth the air extends up about 100 km).

Materials I considered are metal alloy and fiber composites. The best modern steel and aluminum alloys have similar strength to weight ratios, with titanium alloy being slightly higher. Fiber composites have a strength to weight ratio 10x-15x the best metal alloys. Examples of composite material are carbon fiber, fiberglass, and aramid (kevlar) fiber. These materials derive their strength from the fiber part, which is essentially a fabric, and which is held together with a plastic substrate. Carbon nanotube materials (aka graphene) have a theoretical strength to weight ratio of about 20x current composites. Constructing a Banks Orbital, with a diameter of 3 million kilometers (10 times the distance to the moon, or 10 light seconds), would require a new exotic material with a strength to weight ratio 500x better than carbon nanotubes.

Materialstitaniumsteel eglinaluminumbasalt (glass) fiberkevlargrapheneexotic
strength (MPa)10001300414485036205000050000000
density (kg/m3)4420800027002700144010002000
Specific Strength0.230.160.151.802.515025000
https://en.wikipedia.org/wiki/Ultimate_tensile_strength

The relevant properties for space habitat materials are tensile strength and density. Tensile strength (strength under tension or pulling) has the same units as pressure, usually expressed as MPa or psi and is the amount of force per unit area the material can resist before breaking. Density has units of mass over area, often expressed as g/cm3 or kg/m3. Because the space habitat structure needs to support itself, the ratio of strength to density is critical.

It makes sense to build such a habitat out of materials mostly manufactured in space, probably mostly on the moon initially. The obvious material for radiation shielding is simply lunar regolith (sand). Water tanks and other bulk supplies would also presumably be located on the floor/shell of the habitat to act as shielding. Iron, aluminum, and titanium are all available on the moon, although the necessary processes to extract them, especially in sufficient quantity, need to be developed. One particularly interesting type of glass fiber is Basalt fiber, which can be extruded directly from certain types of rock and may be convenient to manufacture on the moon. Chemically, lunar regolith is basically oxygen + metal + silicon, so it may turn out that a single machine can produce both metal and glass (silicon + oxygen) fiber from lunar regolith. There is almost no carbon on the moon, so the plastic part of the composite would need to be manufactured from carbonaceous asteroids.

It’s unclear how to get megatons of nitrogen gas into earth orbit for the station atmosphere, since there isn’t much nitrogen on the moon or on near earth asteroids (oxygen is abundant on the moon). Comets and Trojan asteroids with frozen Ammonia (NH3) are a possible source.

The following table shows the wall thickness of the habitat shell in meters, using various materials. It follows the O’Neill cylinder and Kaplana One papers in using 1G and 0.5 atm of air pressure (42% oxygen, 58% nitrogen). We assume 5 t/m2 of shielding material as well as 1 t/m2 of contents (i.e. people, furniture). We have ignored the mass of thermal radiators, solar panels, and other parts which might be attached to the hub. Calculations use a 1.5x factor of safety on the material stress limit, which is common for aircraft designs.

Namet (steel)t (Al)t (glass)t (kevlar)t (graphene)t (exotic)
Kaplana One0.03230.10160.00850.01140.00088.21E-07
O’Neill Cylinder0.79172.57320.14000.18580.01321.31E-05
McKendree Cylinder1.74680.0015
Bishop Ring4.65140.0033
Banks Orbital41.7442
t is the thickness of the habitat shell in meters with various materials

A steel Kaplana One would be 3cm thick, and a composite version only 1cm. The O’Neill cylinder made of steel as originally envisioned would be nearly a meter thick, but the same cylinder made of the exotic material sufficiently strong to support a Banks Orbital could be only 10 microns thick. The following table shows masses. We assume that the end caps are half as thick as the floor, and that the Bishop Ring and Banks Orbital have 200km tall edge walls.

NameShieldingFurnishingsAir 1atmAir 0.5atmm (steel)m (Al)m (glass)m (kevlar)m (graphene)m (exotic)
Kaplana One2.550.510.080.040.160.170.010.015.00E-049.99E-07
O’Neill Cylinder4,021.24804.251,970.41985.205,252.815,762.16313.51221.8710.910.02
McKendree Cylinder6.65E+071.33E+071.33E+086.65E+072.38E+074.12E+04
Bishop Ring1.57E+073.14E+063.14E+071.57E+071.49E+072.11E+04
Banks Orbital1.41E+112.83E+102.83E+111.41E+112.36E+12
masses are in Mt (1 Mt = 1e9 kg)

The mass of Kaplana One is mostly shielding and contents. A metal O’Neill cylinder would not need any shielding beyond the shell, since the mass of the shell exceeds the mass of required shielding. An exotic matter O’Neill Cylinder would still weight 20,000 tons, or 200 100t SpaceX Starship launches. A steel version would require about 50 million launches for the shell and another 27 million for the air and contents. The masses of the larger habitats are dominated by the air.

The following table shows the wall thickness for the habitats while varying atmospheric pressure and spin gravity.

NameMaterialt (1 atm, 1 G)t (0.5 atm, 1G)t (1 atm, 0.5 G)t (0.5 atm, 0.5 G)
Kaplana Onesteel0.0470.0320.0470.032
O’Neill Cylindersteel1.1580.7920.9020.617
O’Neill Cylinderfiberglass0.2050.1400.2010.138
McKendree Cylindernanotubes2.561.752.371.62
Bishop Ringnanotubes6.804.655.633.85
Banks Orbitalexotic61.0641.7412.898.81
t is the thickness of the habitat shell in meters

The International Space Station is pressurized to 1 atm, while the Apollo missions used pure oxygen at 5psi or 0.33 atm. Pure oxygen atmospheres are impractical for large habitats, since they greatly increase the risk of fires (as in Apollo 1) and require significant adjustment times for people coming from Earth atmospheres to avoid problems with nitrogen decompression sickness. A 0.5 atm pressure seems like a reasonable compromise and has significant mass savings. Reducing the spin gravity doesn’t actually reduce the mass all that much.

Conclusions

We calculated the masses of various space habitats when built with various materials and parameters. Fiberglass or another composite seems like a more practical material for near-future space habitats than metal alloys, with a ~10x reduction in mass. A practical design taking into account more factors like micro-meteorite protection, space weathering, etc, would likely use a combination of materials.

An interesting consequence of the importance of air pressure in habitat design is that rotating space habitats would not consume significantly more materials than large pressurized domes on, for example, Mars. Of course it’s more difficult to access materials in orbit.

Notes

The type of steel I considered is Eglin Steel, which was designed by the USAF and is about twice as strong as the 4130 cro-moly steel in my bicycle, itself about twice as strong as the standard structural steel used for I-beams. Another interesting superalloy is Maraging steel which contains about 20% nickel by weight – coincidentally similar to the ratio of iron to nickel in many nickel iron meteorites (and asteroids).

Carbon fiber (as far as I can tell) has similar properties to kevlar composite, so I’ve only included kevlar in the calculations.

Equations:
t=((Pa+G) r)/(s - a p r)
m=2 pi r^2 h t p

s stress limit of structural material (Pa)
p density of structural material (kg/m3)
r radius of cylinder (m)
h height of cylinder (m)
t thickness of cylinder (m)
Pa pressure of atmosphere (101,325 Pa for 1 atm)
G pressure of contents (Pa, aka kg/m2)
a acceleration due to gravity (9.8 m/s2)

The above equations are derived from the hoop stress equation
s = P r / t
and flywheel stress equation
s = p r^2 w^2
where w is angular velocity (rad/s) and w^2 = a / r

References

Unique Fantasy

I’ve heard a lot about the Indie Bubble, #indiepocalypse, and other doomsday predictions, and watched as fellow developers released good games only to see them crater commercially. This is my assessment of why Reassembly has been a (moderate) success, and what I will focus on for the next game.

My hypothesis is that people play games as a form of fantasy fulfillment and that developers should aim to create games that fulfill a specific unique fantasy as efficiently and powerfully as possible. Anecdotally failure to fulfill a unique fantasy is the most common reason otherwise fun games fail commercially.

Fantasy Fantasy Fantasy

People buy (and enjoy!) games that offer a unique fantasy. A fantasy is like a daydream, a vision of an experience. It can be expressed in a single sentence, a single screenshot, or a couple seconds of video. A game needs to express its fantasy better than any other game or there is no reason to play it. Here are some (obviously reductive) examples:

FTL: command a spaceship like in star trek
Hotline Miami: be a psychopathic twitchy 1980s serial killer
The Stanley Parable: feel like an intellectual game designer
Papers, Please: be a border control agent for an Eastern European country
Fez: re-live 2d nostalgia while exploring the mysteries of the 3rd dimension
Nidhogg: fight in a surreal blood sport for the honor of being eaten by a giant worm
Galak-Z: be a skillful 90s mecha anime protagonist
Minecraft: survive, mine, and craft a sanctuary in a world made of cubes

Some observations:

  • The title of the game specifically references the fantasy, not mechanics.
  • Sometimes the fantasy is experienced by the game characters. Sometimes it is experienced by the human player.
  • Many games interpret a popular fantasy from other media.
  • The fantasy is fundamental to each of these games, and the mechanics and art directly support it.

Reassembly’s fantasy is “build a fleet of spaceships out of blocks and pilot them in combat”. The open world game mode was designed to give players a reason to build different types of spaceships and to generate diverse combat scenarios. Tournament mode gives more reasons to build spaceships. Story was deliberately omitted and the graphics were kept abstract to encourage players to inject their own specific spaceship fantasies into the game.

Work Efficiently

If the game provides a compelling unique fantasy, it doesn’t need excessively high production values. Building the game asymmetrically around your own core competency will insure that it does at least one thing well, which is enough. AAA games need high all-round production values because they are competing for ownership of a few fantasies: be a badass soldier, be a dragon-slaying viking warrior, etc. Don’t try to compete with them! Alexander Bruce’s talk Antichamber: An overnight success seven years in the making excellently addresses this topic.

Let players contribute to the game. This can take the form of Early Access, where players provide feedback and bug reports. Multiplayer games use players to provide high quality teammates and opponents. Games with building allow content created while naturally playing the game to be exchanged with other players (like Reassembly’s Agent/Wormhole system). Modding systems like Steam Workshop allow dedicated players to use external tools to create content. In aggregate players spend many more hours playing games than developers spend building them.

Give players as much mechanical and aesthetic control as possible so they can tune the game to better fulfill their fantasy. Even something as simple as letting players pick their favorite color can greatly increase investment and variety. Component/building systems and procedural content generation provide a huge amount of gameplay variety given how little work they are to implement.

Excessive polish or an extremely smooth tutorial are not important unless they are part of the gameplay fantasy, and can be huge time sinks. Many game developers obsess over making the intro tutorial sequence of their game incredibly smooth and pleasant. This is a form of craftsmanship which I deeply respect, but it is not really what games are about. If the game fantasy does not resonate with players they are not going to play the game anyway. If it resonates, they will be willing to deal with rough patches, even resorting to wikis for forums, in order to experience the fantasy. “Pleasant” and “painless” are not how we want our games to be described.

To reduce costs, Reassembly was developed largely out of my apartment and cafes. My primary costs were housing and food. The game uses a custom engine and vector graphics because I am a pretty good programmer but a terrible artist, and because having lots of totally custom giant spaceships on screen is fundamental to Reassembly’s fantasy. About a tenth of the upfront cost went to contractors, and another tenth to exhibition at conferences like GDC and PAX.

Seek Criticism

Friends and game developers are too empathetic and too polite to provide ideal feedback. They can comment on the game’s craftsmanship, but are usually not the target audience and usually won’t love the fantasy. Side note: Giving polite feedback is ultimately counterproductive, and something I have been guilty of too many times.

Seek feedback from people for whom your game’s fantasy resonates. These are your prospective players. Terms like “core gamers” are meaningless and deceptive. The folks behind SteamSpy wrote an excellent article about how your target audience doesn’t exist. Send free copies of your game in development to receptive people and take their feedback extremely seriously. Reassembly’s initial alpha tester pool found out about the game via gameplay videos on youtube more than a year before launch.

Player feedback was an integral part of Reassembly’s development. Alpha testing began just over a year before the final release (about half way through development), basically immediately after the game became marginally playable. Feedback from these players was invaluable because their only motivation was to shape the game into something they personally really wanted to play. I was able to experiment and learn in a supportive environment, fixing problems as they were encountered through constant feedback. When the game launched on Steam Early Access, we deliberately priced it somewhat high to insure players would be motivated to complain about problems and we would be able to respond to all of their comments. By the time we actually launched, it was clear that there was a market for the game and that the kind of people that purchased the game would enjoy playing it.

It Was A Mistake

I made several technical decisions early on that turned into unnecessary time sinks. Reassembly uses a render thread and a simulation thread, and this dichotomy is extended to every single GUI in the game. In retrospect using a single thread would have been fine in many situations and could have prevented weeks of bug fixing and many annoying crashes. The OSX version of the game uses native Cocoa text rendering instead of SDL_ttf like the Windows and Linux versions – this extra code path duplicates a lot of functionality and honestly the Cocoa text looks worse on non-retina displays anyway.

The Steam Cloud integration in Reassembly is a bit hacky – the world steaming system as designed creates potentially thousands of small sector files, and I hadn’t realized that Steam Cloud had a hard limit of 1000 files until after the game was in Early Access. I worked around the limit with the save compaction system, which periodically stops the world for several minutes and compacts blocks of 256 sector files into a single sector cluster file. This mostly works in practice but has been a constant source of frustration for players, particularly for the most active players with the most sector files.

I wish I had been able to bring on another programmer in the months leading up to the release of the game. I think an extra set of eyes could have seen many stupid bugs and improved the quality of the game in many ways. It may have prevented me from getting quite so burned out after release. I was worried about costs and whether anyone else would be able to understand Reassembly’s scattered code. In retrospect I think it would have been worth while.

Crashes really detract from the game fantasy, particularly when they result in lost data. I wish I had built better automated testing early on. A simple bot that runs around playing the game like the one described in this Talos Principle Talk would have allowed me to find all kinds of problems earlier. I should also have invested in more testing hardware – there have been many GPU specific bugs that could have been prevented by automatically testing the game across a few machines periodically.

Conclusion

I see game development as fundamentally a form of leadership in collective imagination. First, you identify a need for a specific fantasy in your cultural community (man, if we could build spaceships and then blow them up, that would be so cool). Then you build the game, making sure the community is on board and interested. Reaching out to community leaders on youtube and twitch (thanks Deluks, Aavakis, Lathland and others) is the best way to let the relevant community know about the game. A community will ideally develop around and sustain the game (thanks sumplkrum, Camo5, and others). Everyone has a good time building spaceships together.

Tynan Sylvester’s excellent book Designing Games discusses this topic from a different angle and in more depth.

Steam Release Checklist

A curated list of Reassembly bugs that were fixed after release

Every game developer strives to release a perfect game. Many bugs are hard to predict, especially for indies with only a few computers and no experience. I wanted to document some of the things I missed in the hope that other developers will not miss them. These are especially relevant for native (i.e. C++) games which are a little closer to the platform than e.g. Unity games. Almost all of these bugs were fixed during our Early Access period and before the “official” release.

The vast majority of Reassembly work/bugs during the Early Access period related to gameplay elements. The bugs listed here are selected for possibly relevancy to other games and are mostly in the platform layer. A complete list of changes is available on the Steam Announcement Page.

Localization/unicode related

  • Steam usernames are UTF8 and frequently contain ƃᴉzɐɯ∀ st☢ff. Reassembly uses Google’s Droid Sans Fallback and Open Sans Emoji to provide additional characters for rendering usernames, and SDL_ttf or Cocoa/NSFont to actually render the text, depending on the platform. In an ironic twist of fate, the first version of this article was lost because wordpress choked on a U+E10D (rocket) character.
  • The game may be installed in e.g. C:/вещи/Steam/SteamApps/…. so any assets must be loaded using unicode paths. OSX and Linux use UTF8 for everything and are easy to support but Windows builds must convert to wchar_t* UCS2 encoding via WideCharToMultiByte/MultiByteToWideChar or similar.
  • Non-English versions of Windows return native-language error messages in GetLastError/FormatMessage. This means that any logging mechanisms must support unicode. Printf/sprintf on windows support a %ls format specifier for wide character strings but will crash on non-ascii (thanks to Russian alpha tester Krypt for helping me track down this bug).
  • Games that hardcode WASD will not be playable on many Non-English keyboards. Keybinding systems must support unicode keyboard input.

GPU/Graphics related

  • Vsync on/off/adaptive options are probably a good idea. Adaptive Vsync (aka tear control) is a good solution but is not supported on all drivers. Many players have old/slow GPUs that won’t achieve 60fps and consequently would have a bad experience under Vsync. Some players have 144hz monitors and would like to take advantage of them.
  • Should the game launch in a window, in “true” fullscreen mode, or in a borderless fullscreen window (fake fullscreen)? True fullscreen mode has performance and especially smoothness advantages, while fake fullscreen allows easier alt-tabbing. Reassembly simplifies video monitor modeset issues by always using the same fullscreen mode as the desktop, and defaults to true fullscreen.
  • Some people have their desktop set to 16 bit color depth. This may cause OpenGL context creation to fail if you request a 24 bit framebuffer.
  • Significant numbers of Steam users have GPUs that only support OpenGL 2.1. Things like framebuffer objects, floating point framebuffer formats, and various functions may not be available or may only be available through older EXT variants (check GL_EXTENSIONS).
  • Some people have broken or seriously out of date graphics drivers. This is mostly a support problem but…
  • Various GPUs do a poor job of implementing certain OpenGL features. For example, antialiased (GL_LINE_SMOOTH) lines look terrible on many ATI GPUs. Adding video options to the game allows players to work around unforeseen problems.
  • Many people still have 4:3 monitors. If the game resizes UI elements according to the current aspect ratio, make sure to test at 4:3 (in addition to 16:9, 16:10).
  • Many people will disregard minimum system requirements and try to run the game on very old/slow computers. Detecting the OpenGL version and at least popping up a sympathetic dialog is much better than crashing in these cases. Adding options to reduce CPU/GPU usage at the const of graphics quality will also allow many people play your game that otherwise could not.

Hardware/System related

  • Test the game on hard drive systems and SSD systems, particularly if any kind of asynchronous/streaming system is involved. Hard disks can be VERY slow – you may need loading progress bars in places were SSD systems would not.
  • Different gamepads have different polling intervals, button mappings, deadzone requirements, etc. Test at least PS4, PS3, Xbox 360, and Xbox One controllers. With an event based API like SDL_GameController, make sure to poll for ALL events every frame instead of just processing one, or events can get very backed up. Consider analog stick deadzone and smoothing options. Test mouse and keyboard functionality in UIs with a gamepad plugged in, in case stray gamepad events interfere. Make sure to handle the case where a gamepad is detected but initialization fails, or where the gamepad is unplugged/plugged in during gameplay.
  • 5 button mice are common. Make sure you don’t e.g store mouse button state in a bool[3] array with unrelated variables immediately afterwards.
  • Key bindings should probably still work even with caps-lock on.
  • Many Steam users still use Windows XP. Supporting XP/Vista/7/8 with one build involves selecting the “_xp” toolkit in Visual Studio and using GetModuleHandleEx/GetProcAddress for any APIs that were introduced after XP, with fallbacks. Make sure 3rd party DLLs also support XP.
  • Test any scroll wheel functionality on mice AND touchpads. Apple touchpads in particular can generate scroll events at a very different rate than traditional mice.

Steam/Steamworks related

  • There is a 1GB/1000 file limit on Steam Cloud saves. Exceeding this limit results in lost save game data.
  • Many Steamworks API calls (achievements/stats APIs, cloud save APIs, etc. including SteamAPI_RunCallbacks()) seem to grab a global mutex and consequently can block for long periods (multiple frames) of time if another thread is for example writing a large file using the API. My solution was to only ever call steam APIs in a non-critical-path thread.
  • Many API calls (ISteamRemoteStorage::GetFileNameAndSize, ISteamRemoteStorage::FileExists) can be very much slower than OS equivalents. If you call these frequently or depend on their performance, it may make sense to maintain a cache of results.

Crash handling

Reassembly implements a crash handling and reporting system similar to Mozilla/Google Breakpad that has been invaluable in improving quality. Via the Win32 API call SetUnhandledExceptionFilter, it is possible to catch NULL pointer dereferences and other errors that would otherwise crash the program. The program can then collect a stack trace and log file and upload it to a server before popping up an apologetic message and actually crashing. The Reassembly code for this is on github (including a similar implementations via signals for OSX and Linux). We don’t ship .pdb files but include the .dll load addresses to enable associating symbols with the memory addresses in the stack trace.

A CGI script that accepts file uploads can run on any web hosting solution and contain less than 100 lines of code. Compressed HTTP uploads are available through libcurl or dozens of other easy to use libraries.

By collecting (frighteningly) large numbers of crash reports, we were able to triage and fix the most frequently occurring crashes first. It also quickly became obvious that only a small percentage of players report their crashes via email, steam discussion boards, forums, or other explicit methods, and that crashes so reported are not always representative.

This method was very effective for fixing crashing of all kinds, and particularly for race conditions and other bugs that can be hard to reproduce without specific hardware.

It’s very important to anonymize collected log files, including removing information like the current system username which may be present in paths.

Professional QA

Indie Voyage, our Kickstarter/publishing partner, hired a professional QA tester (Chris Watkins) in the month leading up to release. He uncovered a large number of UI, tutorial, and general user experience problems and greatly increased the game quality. In retrospect I would have allocated more of our Kickstarter budged to this area and started sooner.

One of the dangers of Early Access and similar community-driven systems is that players quickly learn the game and focus on advanced end-game features. Players that are put off by the initial experience do not contribute to the community. Focusing development on the end-game is valuable but can leave new players confused.

Final thoughts

Despite my best efforts Reassembly is still not 100% bug free. I’m very proud of how close we have come, and hope the accumulated knowledge is useful to someone.

I did search for a service that would automatically run the game across hundreds of machine and OS configurations looking for problems but was unable to find one. We had a system (actually, at least three different systems) like this while I was at Nvidia for testing the driver and various internal tools and it was great.

Hardware dependent graphics problems are a persistent issue. The cross product of GPU/driver/OS is very large. Actual crashes are great in the sense that they produce stack traces and can usually be worked around directly by e.g. checking GL_EXTENSIONS more carefully or avoiding undefined behavior. Rendering glitches – parts of the scene mysteriously not visible, unexplained z-fighting, etc. – are more insidious. I have not found a good way to debug these besides guessing or buying the GPU in question and hoping to be able to reproduce.

April 2015 plans

I’m back from my post-release vacation and excited to get back to work on Reassembly! These are the things I plan to focus on in the coming month.

New Features

Modding/Workshop support

Reassembly already supports a form of “User Generated Content” through the fleet sharing wormhole/agent system, and many users have had fun with the sandbox or by modifying cvars that control game behavior. I know for a fact that many Reassembly players are significantly better at building spaceships than I am. As a first pass at modding support, I want to expose and document the existing config files and let people publish their changes to the steam workshop, making it easy for other players to enjoy them. The initial mod workflow would be something like this:

  1. In game, convert an existing save slot containing ship designs into a “mod” for new-faction mods, or create an empty “mod”. A mod consists essentially of a number of text config files and spaceship files.
  2. Edit text files to define the mod. This includes adding new blocks or modifying existing blocks, adding or modifying spaceships, changing one of nearly 300 cvars that control various game behaviors, modifying the world generator config file, tutorial or message text, shaders, etc. You will be able to reload config files in-game and view relevant syntax errors.
  3. In game, upload the mod to the steam workshop.

Players will be able to download mods through the steam workshop and then turn them on or off and select precedence (if mods conflict) in-game. I’m excited to see what people come up with! I expect modding support to go through several iterations as we work with modders to expose more functionality and improve the workflow.

Modding support should allow the community to experiment with tons of new content much faster that I would be able to support myself.

Better Fleet AI controls

Better control over AI, both in tournament mode and in the main game, is probably the #1 requested feature. There is already a system in place for generating AI programs based on ship design and faction personality, so most of the work is in UI design. I’m planning to have per-design (this ship should only attack from long range, this ship should always rush, etc.) AI policy plus global fleet policy (everybody stop shooting, attack anything that moves, etc.).

Other Stuff

Localization

A significant fraction of our players are not native English speakers and translating the relatively small amount of in-game text and providing better default keybindings should improve their experience. The game already uses Unicode internally so code changes should be minimal. If anyone is interested in helping translate the game into their native language and/or setting up keybindings, please contact me.

Complete Gamepad Support

Likewise, adding gamepad support to the editor and menus would not be a huge amount of work and would be cool.

Wormhole server / feed

I recently added a search feature to the wormhole feed – there used to be a giant list of users at the bottom of the page but it was getting out of hand. I’ll continue to improve this system as well as the game server which processes wormhole uploads decides which agents populate new worlds.

Bugs / Polish

I’ve received several reports of black screen issues on certain just-barely-supported video cards. There are a few lingering general crashes, and glitches like agents spawning inside of asteroids. The tutorials, as always, need polishing. Some players are reporting trouble downloading agents when starting new games. There are always thousands of little changes that significantly improve gameplay but aren’t worth writing about individually.

Conclusion

I expect this work to take up most of April. Game development is an iterative, community-driven process and we’ll take another hard look at the game for June and see what needs to be done.

February 28th Tournament Results

Thanks to everyone that submitted ships and or tuned into the February 28th tournament! Special thanks to DeluksGaming for joining us!

There were some technical issues getting set up (and some exciting live-coding) so we were only able to run the first two categories. The seeding results are now online:

[Probe] [Interceptor]

Amazigh’s Funnel won the interceptor category and CmdrJohn’s Federation Run And Hide won the probe category.

The twitch streams are recorded on youtube: [Part 1] [Part2]

We will conclude the tournament on March 14th at 2PM PST.

Xcode/Clang gamedev tips

Reassembly was written primarily on OSX using Xcode for compiling/debugging and emacs for text editing. Xcode is clearly designed primarily for developing iOS “apps” and there are several non-obvious settings that vastly improve the desktop game development experience.

My general experience with Xcode has been fairly positive. I think Visual Studio is a slightly better debugger overall but not enough to make me abandon OSX. As a solo developer I need to spend a fair amount of time in cafes and other public places to avoid going insane and as far as I can tell Apple makes the best laptops.

My OSX builds are all 64 bit and have a deployment target of OSX 10.7. The following was written with reference to Xcode 6.1.1.

~/.lldbinit

This config file is read by lldb, the llvm debugger, which is the backend for the xcode debugger. Settings effect the xcode visual debugger window in addition to the console.

Print 64 bit unsigned values in hex and show a summary for vector types.

type format add -f hex uint64 "unsigned long long"
type summary add --inline-children --omit-names float2 double2 cpVect int2 float3

Print a summary for std::pair types (particularly useful for browsing std::map and std::unordered_map).

type summary add --inline-children --omit-names -x "^std::pair<.+>(( )?&)?$"

Print the address, “name” and “blocks” fields of custom type BlockCluster.

type summary add --summary-string "{addr=${var%V} name=${var.name} ${var.blocks}}" BlockCluster

One warning – invalid summary strings often crash Xcode. Restarting the debugging sessions is enough to reload .lldbinit after changes.

Develop No-resource build

For rapid iteration with good performance, I created a “Develop NoResource” target. This build has optimizations enabled (-O3), but no linktime codegen (which takes a long time). While iterating on code remove the main data directory from the “Copy Bundle Resources” build phase to prevent it from being copied every time. As long as you don’t clean previously copied resources will remain in the app directory.

In general lldb is “OK” at debugging optimized builds. It frequently looses the “this” pointer and stack variables but is nonetheless usually useful enough to get the job done. This is one of the areas where Visual Studio really shines, particularly with the magic option /d2Zi+ – see this Random ASCII post. Unfortunately C++ Debug builds, which the debugger readily understands, are usually too slow to be useful.

I have not been able to get the lldb debugger to call inlined stl symbols (e.g. std::vector::size) from the command line in optimized builds. If anyone knows how to do this please tell me.

Platform layers

I haven’t been able to completely abstract OS details into a third party platform independent layer like SDL. Reassembly uses SDL for window and event handling on windows and linux but is pure Cocoa on OSX. The Cocoa programming model is different enough from SDL that some level of native-ness would have been lost, and the amount of code required is not prohibitively large.

In general the Cocoa/Mac programming model is vastly nicer than win32 or POSIX/Linux. It does not carry 30 years of baggage, never refers to 16 bit OSs in in the docs, and provides easy to use APIs for common operations. Using the native API does not take more than a day or two to setup and forever after allows exact control and the ability to debug platform-specific bugs.

Once exception is the OSX Gamepad APIs which are unforgivably convoluted. Reassembly uses SDL to handle gamepads on OSX.

Command Line Builds

Xcode allows building projects from the command line. This is frequently convenient from e.g. emacs or as part of a release script.

xcodebuild -scheme "Reassembly Develop NoResources" \
    -workspace Reassembly.xcodeproj/project.xcworkspace \
    -jobs 4 -parallelizeTargets ONLY_ACTIVE_ARCH=YES

Note that, unlike with visual studio, xcode command line builds do not build the same target as the IDE and so you can’t do a command line build then switch to the IDE for debugging without doing an IDE build. Set the command line config for the non-root project in Project Settings -> Info.

For better interoperability with non-Xcode tools you can set the output directory of the compile .app in Xcode settings (command-,) under the Locations tab (see Derived Data).

Nested Projects

Screen Shot 2015-01-25 at 4.19.36 AM
A little-known feature of Xcode is the ability to add .xcodeproj files as subprojects to your main project file. This is analogous to adding various visual studio projects to the solution file. I generally find it easier to integrate 3rd party code as source vs. using static libs, dlls, dylibs, .so, etc. Creating or adding a separate .xcodeproj per library allows me to specify different compile time options for this code.

Crash Handling

OSX is a POSIX compatible OS which means we can use unix signals to handle Segmentation Faults and other joyous occurrences. Automatically handling game crashes and uploading stack traces to my webserver has been a powerful tool in increasing Reassembly stability, and probably a topic for another article in and of itself. Most of my OSX crash handling code was shamelessly copied from this article on Atomic Objects. Make sure to disable the signal handlers in debug/develop builds or the debugger won’t break on crashes.

The standard backtrace and backtrace_symbols calls work from signal handlers on OSX as expected. backtrace_symbols works even on release binaries and unlike Linux does not require the -rdynamic linker option.

Here is code for printing out some relevant registers on OSX or Linux after catching a SIGSEGV or similar:

    const ucontext_t *ctx = (ucontext_t*)context;
    const mcontext_t &mcontext = ctx->uc_mcontext;

#if __APPLE__
    const uint ecode = mcontext->__es.__err;
#else
    const greg_t ecode = mcontext.gregs[REG_ERR];
#endif
    string msg0 = str_format("Invalid %s to %p", (ecode&4) ? "Exec" : (ecode&2) ? "Write" : "Read", siginfo->si_addr);
    ReportPOSIX(msg0);
    message += msg0 + "\n";

    string mmsg;
#if __APPLE__
#ifdef __LP64__
    mmsg = str_format("PC/RIP: %#llx SP/RSP: %#llx, FP/RBP: %#llx",
                      mcontext->__ss.__rip, mcontext->__ss.__rsp, mcontext->__ss.__rbp);
#else
    mmsg = str_format("PC/EIP: %#x SP/ESP: %#x, FP/EBP: %#x",
                      mcontext->__ss.__eip, mcontext->__ss.__esp, mcontext->__ss.__ebp);
#endif
#else
#ifdef __LP64__
    mmsg = str_format("PC/RIP: %#llx SP/RSP: %#llx, FP/RBP: %#llx",
                      mcontext.gregs[REG_RIP], mcontext.gregs[REG_RSP], mcontext.gregs[REG_RBP]);
#else
    mmsg = str_format("PC/EIP: %#x SP/ESP: %#x, FP/EBP: %#x",
                      mcontext.gregs[REG_EIP], mcontext.gregs[REG_ESP], mcontext.gregs[REG_EBP]);
#endif
#endif
    ReportPOSIX(mmsg);

Threading

By default c++11 std::threads created under OSX have some ridiculously tiny amount of stack space. I use pthreads directly to create threads to overcome this – std::mutex and friends work fine with pthread threads (std::thread wraps pthreads anyway under OSX).

You can also set the current thread name such that it will show up in the Xcode thread view via a non-standard pthread API.

Creating Pthread with increased stack size

#if __APPLE__

typedef pthread_t OL_Thread;
#define THREAD_IS_SELF(B) (pthread_self() == (B))
#define THREAD_ALIVE(B) (B)

OL_Thread thread_create(void *(*start_routine)(void *), void *arg)
{
    int            err = 0;
    pthread_attr_t attr;
    pthread_t      thread;

    err = pthread_attr_init(&attr);
    if (err)
        ReportMessagef("pthread_attr_init error: %s", strerror(err));
    err = pthread_attr_setstacksize(&attr, 8 * 1024 * 1024);
    if (err)
        ReportMessagef("pthread_attr_setstacksize error: %s", strerror(err));
    err = pthread_create(&thread, &attr, start_routine, arg);
    if (err)
        ReportMessagef("pthread_create error: %s", strerror(err));
    return thread;
}


void thread_join(OL_Thread &thread)
{
    if (!thread)
        return;
    int status = pthread_join(thread, NULL);
    ASSERTF(status == 0, "pthread_join: %s", strerror(status));
}

#else

typedef std::thread OL_Thread;
#define THREAD_IS_SELF(B) (std::this_thread::get_id() == (B).get_id())
#define THREAD_ALIVE(B) ((B).joinable()) 

OL_Thread thread_create(void *(*start_routine)(void *), void *arg)
{
    return std::thread(start_routine, arg);
}

void thread_join(OL_Thread& thread)
{
    if (!thread.joinable())
        return;
    try {
        thread.join();
    } catch (std::exception &e) {
        ASSERT_FAILED("std::thread::join()", "%s", e.what());
    }
}

#endif

Naming threads

You can use the pthreads API to set the current thread name, and this name will show up in the Xcode debugger which is extremely helpful. Unfortunately the non-standard pthreads API for this is different across OSX and Linux (and of course windows) – code for all platforms follows, this works in GDB and Visual Studio.

#if _WIN32
//
// Usage: SetThreadName (-1, "MainThread");
//
const DWORD MS_VC_EXCEPTION = 0x406D1388;

#pragma pack(push,8)
typedef struct tagTHREADNAME_INFO
{
    DWORD dwType; // Must be 0x1000.
    LPCSTR szName; // Pointer to name (in user addr space).
    DWORD dwThreadID; // Thread ID (-1=caller thread).
    DWORD dwFlags; // Reserved for future use, must be zero.
} THREADNAME_INFO;
#pragma pack(pop)

void SetThreadName(DWORD dwThreadID, const char* threadName)
{
    THREADNAME_INFO info;
    info.dwType = 0x1000;
    info.szName = threadName;
    info.dwThreadID = dwThreadID;
    info.dwFlags = 0;

    __try
    {
        RaiseException(MS_VC_EXCEPTION, 0, sizeof(info) / sizeof(ULONG_PTR), (ULONG_PTR*)&info);
    }
    __except (EXCEPTION_EXECUTE_HANDLER)
    {
    }
}

#endif

void thread_setup(const char* name)
{
    uint64 tid = 0;
#if _WIN32
    tid = GetCurrentThreadId();
    SetThreadName(tid, name);
#elif __APPLE__
    pthread_threadid_np(pthread_self(), &tid);
    pthread_setname_np(name);
#else // linux
    tid = pthread_self();
    int status = 0;
    // 16 character maximum!
    if ((status = pthread_setname_np(pthread_self(), name)))
        ReportMessagef("pthread_setname_np(pthread_t, const char*) failed: %s", strerror(status));
#endif
}

Github

All of the code above is part of Reassembly and is available in context on my Outlaws core github project – see the os/osx directory for mac specific code.

Youtube

I, Arthur Danskin, the copyright holder of Reassembly, grant you, whoever you are, permission to post or stream footage of Reassembly on youtube, twitch, or similar video sharing sites, in the context of Let’s Plays, reviews, or for other purposes. You may also monetize this footage.

You may additionally include the game music in your video.

Including a link to purchase the game (store.steampowered.com/app/329130) and/or soundtrack (peakssound.bandcamp.com/album/reassembly) is encouraged but not required.

Also, thank you for posting videos. As an independent developer with a very limited advertising budget I rely on this kind of thing to let people know about my game.