more project planning

I'm continuing to evaluate a possible change in direction. I'm going to be partnering up with somebody starting tomorrow which is very exciting! But it necessarily means reconsidering what I'm building. I'm also uncertain of how much I should be writing publicly about it, since now I have more than just my own efforts to screw up!

I will think about a better, more private venue to keep doing these updates. In the meantime, this blog may go dead for a bit.

project planning

Sorry for the missed update yesterday. I'm doing some project planning that isn't so easy to sum up in a blog post. I'm going on a small trip and probably won't update here again until sometime midweek.

windows buildshell fully operational

Today I got the windows version of my buildshell fully operational. It wasn't too bad, mainly because the buildshell just isn't that complicated. But of course there were still a few hangups. I'm embarrassed to admit that I spent a non-negligible amount of time getting the quotesfile working. The quotesfile is completely pointless (you get an interesting/amusing programming quote when starting the buildshell), but it makes me happy.

ubuntu mono env issues resolved (sort of)

Yesterday I found that I could no longer build and run my .net project on ubuntu, after upgrading to 16.04. On investigation this is because my mono environment was also upgraded (to a 4.xxx build). In 4.xxx versions, mono has discontinued support for .NET 3.5. That's awkward, since I'm stuck on 3.5 probably forever (a script upgrade to 4.6 still hasn't even gotten on the Unity technology roadmap).

I tried for most of the day to downgrade to mono 3.12.0. In principle this should be possible. The mono website contains a line about how to do this, which I tried and failed--I added the correct snapshot as my tracked xamarin repository, but I must have mono available in some ubuntu repo as well, because a simple install of mono-complete still added the latest version.

I also found a blog post purporting to explain how to do it. It involved pre-installing a certain set of dependencies, and then installing mono-complete with '-t' to indicate the old version. This also failed in at least two ways.

I wandered down some dark paths trying to find ways to install all the dependencies for old-mono on my machine. Eventually, I more or less gave up, removed all the mono packages, and then installed them back at the newest revision. When I idly tried rebuilding my .net project, it worked!

What seems to be happening is, even though mono 4.xxx no longer supports .net 3.5, it still ships with API stubs for backwards compatibility. Somehow I didn't get them when I did my OS upgrade from 15.10->16.04. But having reinstalled mono (several times, heh), these stubs showed up, letting me build against .NET 3.5, even though monodevelop lists this platform as "uninstalled". Running the unittests works, but it must be going against the 4.5 runtime (as that's all it has).

So I'm back in business, but I don't exactly have warm fuzzies about this. I'd really like to have cross-platform support for developing in this project, but if problems like this keep turning up, the cost may be too high.

laptop setup + xplatform build environment

The linux bug is a thing. There was already a post on the unity forms about it. That's good; I have a workaround so I can stop worrying about it for the moment.

Now I'm working on getting my new laptop set up. I've gotten Unity and subversion install and sync'd down my project. The buildshell that sets things up needs some effort to get fully xplatform, so I'm working on that now. While I worked on that this evening, I discovered that I had broken my ability to build the shared dlls in linux when I upgraded to 16.04. I am now working on straightening that out (it's broken references to the System assembly--even though the assembly exists, right where mono expected it to be).

environment setup + unity bug

In order to further test the Unity bug I saw, I upgraded my local xubuntu install from 15.10 to 16.4. I also upgraded Unity from 5.3.5 to 5.4.1f1. I was still able to reproduce the bug. However, having played around with it some more, I honed in a bit tighter on what exactly I was doing. To add the prefab into the scene, I was dragging it from the Project view, across the Scene view, and into the Hierarchy.

I think somehow when you do this, you get one copy of the prefab added via the Hierarchy, and one that gets added accidentally because of your mouse dragging over the 3d view. Now that I know about this I can work around it.

In other news, my new laptop came today. I have been using a chromebook running crouton/ubuntu as my only transportable development machine (yes, really). It worked surprisingly well for backend programming, but I didn't even consider trying to put Unity on it. My plan is to keep running xubuntu on my desktop, and run Windows 10 on the new laptop. That way I should be able to keep my cross-platform development environment truly cross-platform.

zero day

Nothing to report today. The plan is to continue sorting out whether the Unity problem I'm facing is Linux specific and whether I need to move development back to Windows. However (without any particular excuse), I didn't get anything done.

Instead of an update, please enjoy a photo-diary of the Chernobyl disaster instead.

finished callback pattern, and...unity problems?

First part of the day (the productive part): figured out what was wrong with my unit tests after moving to the new MossDelegate pattern (of course, it was the single-shot change; I hadn't written the unit tests with that in mind and it caused problems that should have been predictable).

Second part of the day: returned to trying to hook up my UI to code in Unity. Only, weird things were happening. I was getting an UnassignedReferenceError on a field in a prefab that totally was filled out (with a GameObject inside the prefab's own dependency tree). I could also see that the GameObject existed in the instantiated prefab, and it showed up in the script's parameters as properly filled out.

Something wasn't adding up, so I started deleting stuff. Eventually I got down to a completely empty scene--not even a camera--and the error was still being thrown. That is to say, a script on a prefab was running and erroring, when--not only wasn't that prefab instantiated, nothing was instantiated.

That seems pretty wrong.

I'm running 5.3.5 on Ubuntu 15 and this is the first time I'm seriously worried I shot myself in the foot by running on an unsupported platform. Things were working so well I got a little complacent. But maybe fun time's over--if this proves to be a linux-specific issue then I think it's back to Windows for me. I will try reproducing with 5.4, and if it still repro's I will try reproducing with the same project on Windows.

I haven't had to fight any real Unity battles yet, so I guess I'm due.

new callback pattern

I wound up needing to build out my callback pattern a bit before continuing with UI. The simplest approach (have the game code publish delegates), had so many problems at The Old Company that I was afraid to apply it here. Instead I wound up implementing something similiar to a solution I made at TOC: a generic interface-based substitution for the C# delegate type.

Since I didn't have legacy code to support, I went one step further and made my custom delegate type single-shot--that is, after it calls its callbacks, it blanks its own callback list.

This serves both for stop-loss (leaked memory can't last past the next invocation of the delegate), and also creates nice patterns for the common case where you're registering for something that is only going to happen once (or at least once per time you want to register for it). The obvious example is logging in--you want to start the login process and then hear that it's done, and that's it; you don't have to write deregistration code and then reason about it to make sure it will be called.

Of course you need a way to handle events that you want to continuously listen for. My solution for that is to have the callback reregister with MossDelegate in a finally block (MossDelegate already needed to be re-entrant safe anyway). Because registering is O(1) without generating any allocations, this seems like a reasonable tradeoff.

wiring and assembly

I'm finally at the stage of wiring up the buy UI to events and static data. Tomorrow I think I should have it actually working.

buy/sell UIX continues

Still hard at work on the Buy Panel. I've drilled down to the BuyEntryDetail dropdown and am slowly filling it out. There's a lot I love about the core idea of this game, but I have to admit it's very data-heavy, requiring these really intense UIs.

UIX Continues

Well, I didn't get as far as I wanted. I think I've mostly solved the "how to use Unity UI tools" problems. Now I'm onto the much more daunting problem domain of "how to make a good UI that holds a lot of information".

Here is what I have so far for my "buy" UI. The goal of this UI is to let you pick goods to take on an Itinerary (a route (possibly multi-city) that you have previously set as a travel plan). The goal is to give all the information necessary to let the player balance risk vs reward ("how much am I prepared to invest? What chance of losing the load to bandits is acceptable?") and also experiment with different route/loadout good combinations. The functional idea is that you get a list of all the things the town sells in a scroll view (the magenta colored area), sorted by sell price value. If you actually want to buy something, you click on the item, and a dropdown appears below it with more information, as well as "Order" and "Cancel" buttons. You can play with your hypothetical loadout as much as you like, and then when you're happy, click "Buy Goods".

UIX continues

Continued working on the Buy/Sell UI. I felt like I really got some traction today, as most of the Unity UI concepts finally started to sink in. Optimistic I will get a fully working version of the prototype done tomorrow. I will post a screenshot with whatever I have.

UIX continues

Nothing much to report today. Kept working on UI design. Repeat all of rant from yesterday.

UIX design

I had been putting it off, but it's officially become hard to avoid working on the game UI, particularly the crucial buy/sell panel. I have to admit, UIX problems aren't my favorite. They are intrinsically messy, and whatever answer you come up with, there's a decent chance most people will be unhappy with it. That said, there's no avoiding it--even to get my core game loop going, I need a pretty developed UI. I've been diving into the Unity docs and begun tinkering, but I still have a ways to go before I can build a dynamic UI with the sophistication I require. First pass of this UI may eat up the rest of this week (with me painfully learning the ropes as I go). Hopefully next UI will be much easier.

front-end + back-end integration complete

I finished the 3.5 downgrade of my backend and got it fully working in Unity. The game client and game server are both hosted on a MonoBehaviour on a GameObject; when you log in, the server spins up on a different thread and the client module automatically logs in to it, returning a PlayerState (there's no persistence, so it's just a newly-created PlayerState every time).

StaticData is likewise working (though still not loading from file)--and hooked up to the MapVisualizer.

This was a big step--all the major pieces are working together now. The next step will be to work on the UI for buying/selling, and the interaction for setting a destination. Those pieces encompass the core gameloop.

downgrading shared dlls to 3.5

Well, I'm dumb. When I started this project I was pulling shared dlls from some earlier work. That earlier solution was building in .NET 4.5. I thought at the time I might have to downgrade someday to 3.5 for compatibility with Unity, but then I forgot all about it. Queue to today, when I actually tried integrating my dlls into my Unity project. Oh yeah, 4.5!

So today was a found-work day. I have a bunch of non-trivial code that relies on 4.5 features, and it took a fair amount of effort to refactor it. In particular, I was using dynamic all over the place in my serialization layer (as followers of this blog from early days might know). I always knew I would have to replace that for iOS compatibility, but I just didn't think I'd have to do it right now.

I was using 'dynamic' because I didn't know how else to handle this problem during deserialization:

1. Create container type via Activator.CreateInstance (let's say it's a HashSet<int>).
2. Now I have 'object container'. 
3. I deserialize the first entry in the HashSet, giving my 'object entry'. 
4. How do I add 'entry' to 'container' ?

I can't cast 'container' to ICollection<object>, because this interface is not contravariant. That means I can't substitute in types higher in the type hierarchy for the generic parameter. This only makes sense; if I could do that then I could violate the type safety of the container, and add, say, a NetworkSocket to my HashSet of ints. 

There is a trick for this, though, as I discovered today. You can cast to just plain non-generic ICollection. That doesn't care what you do. 

Leveraging that, I was able to scrub out all references to 'dynamic' from my code. On the down-side, my ContainerSerializers class, which was already one of the scarier classes in my code base, has gotten still worse. I started from a noble desire to share as much common serialization logic between container objects as possible, but this has led to some pretty weird entanglements. 

I've also had to resort to some kind of hair-raising reflection to unpack the objects, e.g.:

else if( i_gtype == typeof(Queue<>) ) { stype.GetMethod("Enqueue").Invoke(i_collection, i_obj ); }

--because ICollection has no Add(object)--I can't just cast all these ICollection containers to their common type and then do an add. I have to get weird. Hope that works on iOS. 

worldbuilding, static data design

Finally back from my trip. A bit of a slow start returning to my work routine. Today I worked on static data design, outlining the static data files I think I'm going to need, and what's going to be in all of them. I also started grappling more earnestly with questions of scale--how big the world is, how far apart things are, and how far it takes to get around.

I also found a flash game online that's built on the core trading premise I'm pursuing. I thought it was rather weighed down by the bolted-on addition of a tactical RPG, as well as some truly grognardian levels of simulation detail. It was also all but impossible to figure out what to buy to make the most money, because the interface doesn't really help you--you need to actually wade through some NPC conversations to figure out what people are buying. But underneath all that, the core act of buying something and then selling it for a profit felt pretty gratifying, which made me happy.

worldbuilding

Writing from Maine. Sorry for the missing post yesterday--I did get back to work after a fashion, and managed roughly half a day. Since I've always found the summer cabin conducive to writing, I spent time working on worldbuilding. I am building out one of the six regions of the continent of Almaia, the Farahdi Peninsula. It's a little daunting to think about how many place-names I'm going to need, but I'm trying to do just a bit at a time (and the friends-and-family launch goal is to have just one region open).

I have a bunch of chores to do today, as well as travel, but I'd still like to get in another half-day.

integrating unity project into tree

More than a month ago now, I was working on a simple Unity project called MapTest. Basically it just rendered a map texture and then placed some "town" icons down on it via script. The organization of the project was very prototype-y (i.e., there was no organization). Now has come the time to integrate that humble beginning into my main project branch. I spent most of the day cleaning up this project and getting it working from my build shell. I also did some planning for the next several steps I need to do hook up the unity project (now called clientviz) to the backend I've already written.

GameFrame is fully functional as of this morning. I was able to do a full integration test successfully. Because the client and server were running in the same process, I could do a whole series of PlayerState operations and then compare the client and server PlayerStates to confirm their equality. This was a little tricky because of comparing a CPlayerState and SPlayerState (server and client specializations), but it turns out the Kellerman comparer has the right config options to support this.

I will be traveling tomorrow but hope to get a 1/2 day in. Then I will be off until Monday 29th.

GameFrame

Worked on automated full-lifecycle tests for the game. Someday GameFrame will be used to do client-server integration testing, but right now it's running a local server, and exercising all the client->server events that currently exist. Still a couple problems to work out, but close. Once this step is done it's back to Unity.

CVClient, GameFrame

Well, it's actually called CaravaneerClient now, not CVClient. Turns out that naming a class CVClient inside a namespace CVClient is a bad idea, even though the language lets you do it. In any case, it's stubbed out to a fair degree of maturity. I'm now starting to rough out GameFrame, the command-line test tool that does integration testing from inside the Mono Solution. The hope is to have the server and client parts of the game all talking to each other, in a way suitable for batch-mode testing, all without having to touch Unity.

CPlayerState

built out the second PlayerState request object: AdminGrantRequest. Added CPlayerState and started work on the client side of the transaction pipeline. 

I'm thinking about how best to build a request/response pattern on top of the networking layer I wrote. That layer is entirely object based (just streams of objects in each direction). I think I'd like to keep it that dumb, and have ClientRequest /ServerResponse objects that contain the necessary state to provide callbacks for specific requests on the client. 

SPlayerState, mossbin

Today I straightened out an issue with PlayerState and SPlayerState that I was unsatisfied with. As written, SPlayerState extended PlayerState compositionally. It did this because of serialization. One of my original design choices for mossbin was that serialization was heritable. that is, if you had:

[WritableClass]
PlayerState {}

SPlayerState : PlayerState {}

SPlayerState would inherit the WritableClass attribute and implicitly become a serializable class (with the exact same fields as its parent).

But now imagine on the server you have this:

[WritableClass]
LoginResponse
{
   [WritableField]
   PlayerState m_state
}

If you fill out m_state with an SPlayerState, the server will write out an SPlayerState--even though that type is defined in the server dll! Obviously the client won't know what to do with this thing. For this reason, SPlayerState contained a PlayerState via composition. However, as I'm about ready to build out the client side, including creating a CPlayerState, I began to feel the limitations of this pattern. Actual inheritance would be tidier for my purposes.

So I changed my tune. Now [WritableClass] is NOT an inheritable attribute. In addition, if mossbin encounters a [WritableField] that is a base-type reference pointing to a non-serializable child, it will automatically walk up the inheritance tree, looking for a base type to serialize as. So you CAN assign SPlayerState to the PlayerState reference, and have it serialize as a PlayerState that the client can understand.

In a way, this is its own sharp edge (I can imagine someone--maybe future me--being bamboozled by the fact that I had a class with m_parent->child on one side of a serialization, and got back m_parent->parent on the other side). But it's super-convenient for this server-client relationship, where specializations on both sides want to communicate by their common base class.

This actually created a new missing feature I had to document. Say you have:

[WritableField]
Parent m_parent; //(points to type Child after instantiation)

You might want to serialize this as a Parent in one situation (say, sending something down to the client, but actually serialize it as the Child that it is in another (say, the server saving its own specialized state into the database). This will entail a new property on WritableField, and some extra logic in mossbin.

I'm still not sure writing my own serialization layer was a good idea, but it sure is fun being able to tune things at this level of detail.

TownSnapshot

The big thing I did today was to get town snapshots working. Basically the thing I talked about yesterday. There's now an EnterTownRequest object, and when it goes through, the server caches the state of the town's market off on the PlayerSession (this is all a bit of a puppet show, because there's no time-variability in prices yet).

I also stubbed out the cvclient dll. For a long time I wasn't even sure I'd need one, since the interactions on the client with game state didn't seem too rich (rather, the unique, clienty bits). But I eventually talked myself into it. I think it might be a good design goal of the Unity visualization layer wasn't even allowed access to the shared-dll namespace (even though of course it's linking the shared dll). That would be some good concept-isolation between the game and the visualizer, if it could be done.

EconomicSim, CitiesData

Built out placeholders for EconomicSim and CitiesData. Beefed up WorldState. Added another worker thread for WorldState to run in (CVServer now manages two; one for world state and the other for the session manager handling client connections).

I realized I hadn't really thought a part of my design all the way through. When you bought or sold a good, the idea was you'd just consult static data to find out how much the good was (which would be the same between server and client, because that state is all server authoritative). But my design for good prices is time sensitive (and potentially sensitive to other variables). The client needs a snapshot of state from the server that it will know is good for a certain amount of time, and then that needs to get threaded through to all PlayerState-modifying requests. To do tomorrow.

I'm really hoping to return to Unity soon--it would be nice to start hooking up all this backend to something graphical.

PriceList, EnterTownRequest, EconomicSim

For the prototype, I need the server to return a PriceList of buy/sell data when reaching a town. The prices of goods are governed by the EconomicSim (a class that doesn't actually exist yet)--itself a part of WorldState. The WorldState simulation is intended to run on its own thread and expose two interfaces: a read-interface that is thread-safe, and a write-interface that is basically just a ConcurrentQueue for receiving update requests. 

I've started thinking about how to do the thread-safe read interface. When I first started thinking about this problem, I was imagining some kind of double-buffering solution: two copies of WorldState, one used by the readers, and one by the WorldState simulation: after every WorldState useTime, it would swap the two with an interlocked operation. 

That's not quite adequate, though. If the WorldState usetime runs again while receiver threads are still using WorldState from the last tick, you could get an unsynchronized conflict (the read-only version has now become the writable version, and is being scribbled to). I'd need to some explicit locking to prevent this, which defeats the purpose. Maybe I should just use a ReaderWriterLock and design WorldState's update pattern in such a way that the updates are very brief in duration (e.g., doing all the complicated reasoning outside the lock--knowing that our thread is the only thing that can modify the state anyway). 

CVServer, SessionManager, staticdata stubs, LocalizedString, PostDeserialize method

I stayed up later than I meant to. Almost got a workday in despite starting around 7. Lots of pretty easy stuff. I stubbed out the main server class and the SessionManager class, and wrote the logic for kicking off and joining the SessionManager's worker thread. Then I started stubbing out static data. In doing so I realized I needed to stub out localized strings. In doing that I realized I finally needed to implement PostDeserialize in my serialization layer (the idea is that after an object unpacks, the serialization logic will call a method tagged with the 'PostDeserialize' attribute to let it do final setup; LocalizedString will use this to convert from the serialized string hash Id to the actual localized string literal by referencing the localized string pool). 

I finished PostDeserialize (with unit test). The rest is mostly stub. I'm very deliberately resisting the temptation to actually build out the resource model since that was explicitly a non-goal for the prototype. So all the static data is going to be populated with some hard coded values. 

PlayerSession, static data

Pretty simple today. I finally worked out all my remaining failing unit tests and got a full lifecycle of the PlayerSession working. It turns out the serialization bug I thought I understood was a little more interesting. verbum sap: if you're using Type.GetField, you won't get back private fields from parent types even with the BindingFlags.NonPublic attribute set! That doesn't make a lot of sense to me; they're still private fields in the type, and you can see the other private fields. Whatever--easy enough to fix once understood.

Now I'm on to blocking out the pattern for static data, although I think it will be a while before there actually is any (i.e. files, instead of things hardcoded in classes).

more PlayerSession, unit tests

Not much to report today; was traveling back from Maine into the hot sticky weather of Boston. I worked a bit more on PlayerSession and found a new bug in the serialization layer.

PlayerSession, unit tests

Sorry for the lack of post yesterday; I took it as a vacation day and went on a beautiful hike, and to a concert in the evening. 

Today I continued work on PlayerSession, the core class for managing player's connection lifetime. I've come up with some pretty intricate unit tests for it, including a full life cycle (connect, do a PlayerState transaction, and shutdown), and several failure cases. I discovered a problem in NetConnection; I had done some very server-centric error handling in receiver thread. To do that error handling I had had to mix in a more specialized namespace that I had intended to keep separate. Turns out that was wrong anyway--the logic I wrote wouldn't make any sense on the client. One of those cases where, if I had done it properly the first time, the system would have "just worked". 

Up to 105 unit tests now. 

PlayerSession, SPlayerState, unit tests

I got significantly closer to completing all code necessary for the full client lifecycle today. That will encompass logging in, making a simple transaction (buying or selling some stuff) and then logging back out again. I've added the server dll and its first class, SPlayerState, a compositional child of PlayerState. It contains the server's message handling logic for PlayerState requests. There will be something very similar on the child where, if you want to, say, buy something on the PlayerState, you call the base class (doing the actual buying locally), and then do the logic of creating a TransactGoodRequest object and sending it to the server.

I've also been building out the logic for rolling back from a ValidationException. Most exceptions cause a client disco, but ValidationExceptions (which are thrown by game code when an update to the PlayerState doesn't pass muster) send back a whole copy of the "good" PlayerState. My hope is to hook this up so that it resets the user to a known good state in a much less traumatic way than a disco (maybe showing a wheel rolling backward or something, to fit the caravan theme).

I'm writing this on my laptop from the family place in Maine; I took a 2 hour lunch to go windsurfing. Liking the self-employment so far.

cork work; message handlers

Nothing to exciting today. Blocked out the server dll and implemented the PlayerSession class. Threaded in the message handlers that move message objects out of the NetConnection and then to the PlayerState. By design, all this happens in the receiver thread, so replying to a request doesn't entail crossing any thread boundary.

back to work; core request/response classes

Finally completed my house move and resumed regular work. Today I continued work on the underlying ClientRequest / ServerResponse, message handling, and exception handling pattern. In doing so I think I've bumped up against a bug in my underlying serialization layer, which is exciting because I have a lot of unit tests in there.

moving

Just worked on my apartment move today. I think I'm going to be unable to accomplish anything of use until this Sunday at the earliest. I will probably skip on updates until then.

core game classes

I started blocking out core game classes, and worked on the synchronization strategy for PlayerState (how it will update on the server with validation, and how it will roll back on validation failure).

Because of my upcoming move I sold my AC yesterday. It's been pretty muggy today--hard to adjust. Looking forward to going now.

PeekableStream, AsyncMemoryStream, NetConnection

Yesterday and today.

I've pretty much got my application transport layer functioning at levels acceptable for the prototype. I can hand off an object to my NetConnection object, and it will asynchronously serialize that object into its outgoing AsyncMemoryStream. The other side can then reconstitute it; the actual deserialization also happens on a worker thread, so all the main thread has to do is just pop off an object reference from a queue.

Yesterday things were not going well. Despite all my careful unit tests, when I did a fully assembled asynchronous test of sending objects, something was causing the stream to be garbage on the other end. Of course I was sure it must be a multithread problem--was I somehow kicking off two pool threads that were doing sends simultaneously? I fell asleep racking my brain about it.

This morning I woke up and had the answer. It wasn't a MT problem at all. I had misunderstood the expected behavior of a stream. If you tried to read 100 bytes from my AsyncMemoryStream, and there were 51 available, it would give you back 51 (the stream interface returns how many bytes are actually read). But that was wrong. You should only do a short read when you hit EndOfStream, and that's not the same as "oh there just aren't that many bytes right now".

I couldn't understand why I wasn't getting IOExceptions out of the BinaryReader that wrapped the stream, though, since that what happened when I did a trivial test, calling reader.Read(int32) with only 2 bytes in the stream. But it turns out the behavior of BinaryReader is inconsistent: if you read a string, or an int, you will get an IOException, but you try to read 'n' bytes, it will short read without error. This is explained in the MSDN docs, but I missed it. I have added in more unit tests around this.

PeekableStream

Almost finished PeekableStream today. I think I'm one unit test away from completion. I thought this would be a 3 hour class, instead most of two days! Shows what I know.

NetConnection, PeekableStream

I was able to finish up my AsyncMemoryStream today, and with it get my NetConnection class working in unit test. Pretty neat to see my application transport layer shuttling objects around over streams. However when I tried sending a non-trivial compound object, I immediately got a failure: the Seek method of my AsyncMemoryStream was being called, and wasn't implemented.

I could implement that method, but then I remembered that I shouldn't; you can't seek on a NetStream, so no seeking on AsyncMemoryStreams either. Being able to Seek (at least enough to peek 4 bytes at the end of a stream) is a required feature for my serialization layer. I remember (months and months ago), I decided I'd implement a PeekableStream to accommodate this whenever I got to the level of actually using my serialization code.

Well, the day has come. Spent rest of day working on a PeekableStream. You compose it with non seeking streams, and it provides a limited ability to seek back from END (just enough to peek something like a BinTag off the end).

AsyncMemoryStream

More fun with my MT memory-based stream. Threaded systems are pretty exciting to write, in a "riding a motorcycle naked the wrong direction down the highway" sort of way.

I finished the async Stream methods (BeginRead/BeginWrite) and wrote some good unit tests for them, although in retrospect I'm not even positive I'm going to use them.

I'm adding an accessor now that lets the user know if bytes are available to be read on the stream. Unfortunately I'm using a volatile bool. I understand volatile to be the spawn of beezlebub, but I'm not sure what a better approach would be. I can't just ask for the length of the stream, because that takes a monitor lock, and that could block for the whole duration of a Write--rather defeating the point.

AsyncMemoryStream

Well, that was fun. I haven't stayed up late hacking in a while.

The plan for the prototype is to be client-only, no persistence. But of course I can't help thinking about the future. I wanted to make a fairly "real" feeling application transport layer that operated on the Stream level. For now, the client's input and output streams will just be hooked up to a server module that is running in-process.

To make that work, though, I needed a thread-aware Stream that acted like FIFO byte queue, and I don't think there is anything like that out of the box in C#.

So here I am working on one. Most of the synchronous operations are in place, but I still have some unit test failures around cases where the internal ring buffer needs to grow. I'd really like to get the async Read/Write methods working asynchronously too (via deferring the work to the ThreadPool). I think that be a good "broom" for async bugs that might not otherwise show up until much later.

workspace layout, core classes

Mostly finished assembling the new project workspace for my prototype. Design work on using my mossbin serialization format in this project, and on what some core of the core NetRequest / NetResponse classes might look like (even though the prototype is client-only, the architecture is still client-server).

planning, enums

Sorry for the long silence; I was on vacation this week, returning to work today.

Today I finished implementing my solution for enums. I decided to go with semantically self-contained enums, which is an idea I've found attractive for a while. Basically, the enum is a 64-bit integer. The integer value is formed from packing 12 characters together using a special 5-bit encoding. That's a little better than DOS filenames, even! This ducks a lot of annoying questions that come up with enums, like how to prevent re-enumerating a value, and how to coordinate enum value assignment across multiple developers.

I worked on this now because I'm almost ready to start mocking up game data. Pretty excited about that.

mapviz

Continued working on map visualization in my current prototype project. Not a lot to show for today.

prototyping

I felt strongly enough about the state of my current writeup to start prototyping. In a slightly creepy way, it reminds me of the project at The Old Job (at least in that it has map, and would work well for async multiplayer and mobile). I guess you can never really escape your past.

I got the basic interaction in the map screen working, and I've drawn out a set of three UI screens that I'm going to try to build as part of the prototype. That should be enough to understand whether there's any fun to be had here.

scenario design, art

Yesterday and today, kept plugging away on scenario design, and spent a bit of energy working on a basic blockman to use for testing animation. While I like the scenario I've been writing up, and I think it's promising for the sort of narrative/puzzle game I wanted to work on my first time out, I'm worried that even on paper it might be too expansive. On the other hand, this is the third scenario I've worked on, and if I keep going like this I won't have much to show besides a large pile of half-written game summaries.

I will settle down on something eventually. Small is hard.

scenario design

Not too much to report for today. More notebook scribbling / scenario design work, mainly. I will try to get some good productivity in on Saturday to make up for it.

shaders

Not much to report today, between getting my car in for service and seeing off a former coworker, who is soon to be starting an exciting new gig of his own. I did experiment more with different surface shader options, and got vertex-colors working with the PBR lighting option (not hard). I'm running out of excuses before diving back into scenario design! That will be tomorrow, I think, hopefully including some level mockups in Blender.

movement model + asset import

Got some good things done today:
  1) Finished the code for a basic 3rd person movement model. It handles step-ups and edge sliding correctly. It should be very simple to extend to support "max walkable slope" and a "stick threshold" below which you just run into the wall and don't slide along it.
  2) Imported a walkable mesh from Blender into Unity. I created a simple platform and figured out how to get into Unity and make it have physics. I used Unity's default physics mesh generation, which may not always be ideal. A future step is to figure out a good authorial way to create a physics mesh in blender, based off the render mesh, and then import them together.
  3) Get a primitive surface shader working. I want to be able to use vertex painting to basic coloring, and for that I can't use the Standard shader, which doesn't support vertex colors as far as I can tell. I'm hopefully close to making this go, and in the process I think I'm going to be a lot that will be useful later.

movement model

Yesterday and today, spent some fun times playing with movement models and ideas for how to describe walkable areas in a game space.

environment art

Another day playing with environment art / trees.

environment art

For the last couple of days, I've been working on environment art, basically trying to build a small 3D level in Unity. It's slow-going, but I feel like I'm learning a lot. It's all a bit of a digression, but the idea is that it's going to teach me how to make a visualizer that could look good and be visually distinct, while still falling within the project's constraints. Also, I've learned a lot about Unity's tree-tool :-)

scenario design

Yesterday and today, continued plugging along on scenario design for my current concept. The idea is  a fairly simple graph manipulation puzzle, where you need to untangle a graph to remove all its cycles, dressed up in a narrative. But coming up with the right level of complexity for the underlying puzzle has been tricky, and I'm still working on it.

art, scenario design

Yesterday I spent mostly in blender, working on landscape tiles for a landscape system that I may not end up using. I learned (in some cases relearned) a bunch about how to basic box and extrusion modeling in blender, though.

Today I spent brainstorming more about exactly what I'm building. I feel more strongly about making some kind of "pet the sloth" experience, but a scenario that fits that limited scope, that would still be satisfying to build, is a pretty tight set of constraints. Toward the end of the day I hit on an idea I like, and I'm still notebook-scribbling on it.

viz+art

Another experimental day playing with art tools. I spent the morning trying to do perspective drawings of environment art in Krita, and the afternoon playing in Blender. It doesn't feel like I'm making much progress, but I think this phase is necessary. This is certainly one of my areas of weakness, and the goal isn't really to create production art, but to somehow articulate the idea of a particular 3d space in my head inside a tool. I've had fun doodling simple 1 and 2-point perspective spaces in my notebook, and I think once I've gotten to the stage of capturing that in blender it will be pretty neat.

viz

Still experimenting with visualization solutions. Yesterday I got a little C#/OpenGL program running, with enough functionality to get a sense for what it would take to implement a basic 3d visualizer. Today I got Unity set up on my linux machine. This was surprisingly easy--I just took the latest .deb file from the linux thread on the unity forums, let it install its large number of dependencies, and then there was unity. Pretty neat. Unity certainly deserves serious consideration. It has downsides, but it's the devil I know. And it runs in linux after all :-)

viz

[viz][5hrs] I went from a hacky DLL-path finding solution to a rather nice one. My visualizer dependencies are nicely contained in an sdk directory, and my buildshell python script sets up all the necessary environment variables (LD_LIBRARY_PATH, MONO_PATH). I experimented with OpenGL.Net bindings, but there's not a lot of information on setting them up on linux, and it definitely seems like I'm missing a few pieces (create the OpenGL context? set .config files so the dll can actually find the SOs?). So...back to openTK for now.

viz, dll hell

[moss][7hrs] Turned my attention fully to prototyping the visualizer today. After much internal debate, I decided the first-pass visualizer will be a managed process that implements its own custom renderer via managed bindings to the OpenGL API. The goal of this is to get a 3D scene driven by a simple iteration of the visualizer protocol up and running, and to give me an opportunity to dip my toes into graphics programming.

The final form of the visualizer is still very much TBD--certainly it's the area I have the least technical background in, and I'm feeling my way as I go. After much DLL-hell, I've reached "spinning cube demo program", with SFML and OpenTK as the dependencies.

unittest + visualizer planning

[moss][4hrs] Not much accomplished today. Finished hooking up all existing unittests and began planning for visualizer.

unittest

[moss][7hrs] Got unittest running in new build shell, and scanning most of the interesting methods in mosscore. Whenever I actually implement unit tests, it never fails to shock me how many problems they find. In fact, it looks like there's a problem with polymorphic serialization, which I know was working at one point. I will sort that out tomorrow. Then maybe it will be time to start thinking about the visualizer!

buildshell

[moss][4hrs] slower progress today. Late start. Added a rudimentary build shell, and some basic logging and arg parsing features. It's cool to be able to run unittest from the build shell and see a bunch of stuff getting tested--even cooler when unittest will run the more interesting serialization unit tests that live in my core dll.

edit: I will be off tomorrow and Monday.

housekeeping

[moss][6hrs] Got unittest harness up and running. It uses reflection to scan a known set of assemblies for participating static methods, and then calls the methods. It then does all the normal stuff with the results (displaying success/fail with pretty colors, displays any exceptions that occured). It also tracks execution time. My ambition is to have the unittest module keep a local record of execution times (for tests that signaled that their times were "interesting", and not highly random or something), and then warn if that time changed by more than some % delta. 

Still building the house I want to live in...

serialization layer + housekeeping

[serialization][3hrs] Finished support for Dictionaries. Integrated a nuget package that does deep (reflection based) comparison of arbitrary objects to see if they are value-equal. Used that to add some tests of some very hairy compound containers. 

[environment][2.5hrs] Started putting together project directory. Ported over serialization from its test project. serialization will be the first feature, and a testground for exercising some of the facilities of the project: unit testing, code validation, and the core mosstool for doing things like updating the TypeMap. 

serialization layer

[moss][9hrs] Wow, the day just flew by. I'm enjoying the pleasant illusion of rapid progress. I wrote a whole great steaming mess of code, anyway.

notes from today's CL.

1) BinTag helper methods to facilitate making a BinTag directly from a Type
2) Added Primitive flag to BinTag to make error-cases better.
3) ContainerSerializers. Add support for:
   a. Stack (requires custom writer that reverses order)
   b. Arrays
   c. Nested containers (required whole new methods for recursively writing all the
      BinTags necessary to fully specify a type and then read them back.
4) Primitive/Container Serializers initialization flow entrusted to TypeMap.
   This feels better.
5) Removed createType from SerializableType as it was not that interesting. On as side-note
   container deserialization now presizes to correct capacity!
6) SerializableType now remembers the short name of its source Assembly in file state.
   It uses this in concert with the new TypeMap assembly cache to locate the Type
   at runtime. This turned out to be harder than I thought. Type.GetType requires a
   fully qualified Assembly name (unless in the local assembly or mscorlib). But the
   FQAN has the version embedded in it! I would like to follow semantic versioning rules,
   which means this string will change a lot. My solution is to store only the short
   assembly name, and then search all assemblies for the assembly that matches it (whatever
   the version is).
7) Removed initalizeAssemblyState from TypeMap. I've rethought the flows for TypeMap
   initialization. When creating ex-niliho, you call initializeFromAssemblies, and
   pass in an array of Assemblies to scan for participating types. When updating,
   you will call updateFromAssemblies, which will take both the TypeMap file and
   the Assemblies array. When using at Runtime, you will initialize via initializeFromFile,
   which will take a parameter specifying whether runtime state is mandatory.

serialization layer

[1hr] Wrote up engineering design goals for MOSS, as well as user experience goals for a hypothetical game that might sit on top of MOSS.

[3hrs] Got container serialization working for Lists. All other Collections should follow trivially. I think a new method-pair will be necessary for conventional C# arrays. Then associative containers, and I'll be done with this part. The hard bit was of course coercing the C# type system to not complain about all the vague things I was doing. For instance, I had:

object entry = Serializers.deserializeInternal( i_reader ); // i_reader is a BinaryReader
container.Add( entry); // container is a dynamic created via reflection.

This doesn't fly. The runtime can figure out that the List indeed has an Add method, but it's detecting as Add<int>, and you can't pass an Object to Add<int>, even if the Object happens to be an int. I don't think there's any general way to do a cast like:

Type someType = object.GetType();
(someType)object;

Solution? Make entry also be dynamic. Duh. Just keep making things dynamic until it works. Should have thought of that immediately.