‘New Reality’ Development

This blog has moved, please update your bookmarks and feeds:

http://james.black0ps.com/

The new server is faster in every way, and should be more accessible than from this domain.

Thanks!

The weekend was good – I revisited some graphics programming, and after all this time, I had some catching up to do. Admittedly, I forgot exactly how to translate and rotate in XNA, and had to revise over an old camera class to spark my memory. It’s all good now though. I started the development of a sun – it moves across the sky, and, if you set the time of day, it will re-position the sun based on the time. This is no difficult feat though – there are 84,600 seconds in the day. The sun will be moving in a complete 360 degrees. So we convert the values from 0 – 84,600 to 0 – 360, and feed this function the time of day (in seconds).

The sun also does not behave so predictably colour-wise, however. So far, the sun has a static colour, which would be good for mid day, but bad for sunrise and sunset (and night time!). So for this, I have three colours – dawn, noon, and dusk. The time of day is used to interpolate between the colours as the sun moves across the sky.

Now comes something slightly more advanced. If the user wishes to position the sun manually, we need to do some collision detection. We use a bounding sphere the same size as the sun’s theoretical sphere (the radius is the distance from the world origin),  and then, with basic picking, see where the mouse’s pick ray intersects the sphere. The resulting coordinate is the new location for the sun, or it can be drag/dropped to avoid accidental replacement if the user accidentally clicks the sky. The value of the sun’s roll (the sun orbits the Z-axis – we have so far used +z for North, +x for East) is then converted to seconds in order to calculate the colour.

There are loads of concepts being queued up for the graphics work. The blessing of a hard but fun, rewarding task is also a problem elsewhere. Skinning models, calculating light, calculating procedural grass and terrains absolutely eclipses PHP work. Programming is fun when it is challenging, and PHP offers very little challenge – just monotonous coding, usually in slightly different ways. The biggest reward of website development is seeing people use the functionality and get something from your site. The best part of graphics is everywhere – you are creating your own worlds. There are so many concepts to get through that it offers a constant challenge.

Maths, in my opinion, is the greatest strength you can possibly bring to graphics programming. It is everywhere. It is complicated. My maths from school was pretty bad, and I can safely say that most the maths I’ve learned has come from graphics work. It is getting more understandable, but I really do wish I got into it back before things were so critical.

Before, I expressed a problem I had with designing a website. Now, I am writing in a better, more confident mood about it.

Actually, the hardest part was finding colours that work together. A problem here is that it is relative. But a counter to that is that if someone is looking for particular content, they won’t sit and review the site’s theme. But again, to counter that, first impressions mean a lot.

The obvious is for the theme to look clear, so that the individual can find what they want as quickly an efficiently as possible. That would be one good first impression. No one likes visiting a site they haven’t been to before, and spending even minutes trying to find what they are looking for.

People who regularly come back to the site also don’t want to sit in front of a theme that makes their eyes bleed. For the background of the site, it was a tough choice – to go against the site’s main use of blue and peach, I tapped into green’s range, ever so slightly. The result was less taxing on the eyes than the same almost-grey-but-purple shade.

I’m also suprised at the detail added by simple drop shadows and rounded rectangles. They are small details, but they make a suprising difference. When using the old theme, the site feels old. The new theme makes it feel fresh, and also more robust, as if the quality of the theme reflects the quality of the functionality.

These are still beginner approaches, but they do make a difference. This has given me an interest in image editing to some degree. Finding my way around GIMP and Photoshop would make things a lot nicer when it comes to editing textures for the game engine.

Can’t Wait for Christmas

Ah yes, Christmas is not too far off. But i’m not looking forward to it for the presents and feasts – my partner in crime and I have decided to set aside a few weeks to work full time on the game engine. The result should be a significant part of a small RPG engine. That is, we should have procedurally generated terrain, with realistic grass, omni-directional shadow maps, a nice particle engine, an inventory system, convincing water, vegetation, and detailed texture mapping on the terrain (with the vegetation following suit). Anything less than the majority of those features would cause a bitter resent for the time and how it was used.

29th September. Time’s ticking by, and the site has really taken shape. Is it ready for the public? Can it survive the public scrutiny? Are the bugs ironed out?

Time will tell.

Either way, the site has been released at long last.

To give it a nice, memorable date, I spent the remaining 5 hours of Wednesday making sure everything was going smooth. The site was then uploaded, the database cleared of test data, and the countdown began – the idea being to initialise the site as close to midnight as possible. The site’s “birthday” then being 1st October 2009.

As I type, there are now 11 events. After hours of data inputting, I came across a rather serious dilemma. You see, Britain is divided into counties. These counties have changed over the years, some changed during another change. We are now left with some nice large, general counties, some sub-divided counties, some created to help the Post Office, others created to serve lieutenancies. The trouble here is that I don’t know which county sets to use. One idea is to use all of them, but different sources of information provide conflicting data on which county’s serve which purpose.

In the end, I didn’t see much harm in adding all county’s. Except when you try to find all towns/cities in, say, Wales, and the data there is different to how you know it. Take Swansea – it should go in the County of Swansea, yet Yahoo’s directory lists it under West Glamorgan. Which is correct depending on how you look at it, but will someone searching via county get the results they expect?

In an ideal world, people would be fully aware that Swansea (and County of Swansea) is in West Glamorgan, which is in turn, in Glamorgan – the original county. But will people select those? That’s not even the entire point – the point is, this is adding a complication to what should be a very quick way of getting information. If someone is trying to find Singleton Hospital, then one would assume they would select Swansea as a county. But in this case, the user would not find the event, as the event is listed under West Glamorgan.

So here comes the solution – parenting. County of Swansea is added as a child county to Mid Glamorgan, which is in turn a child county of Glamorgan. Selecting either in this relationship shows the necessary events. But this adds a problem that I, as an individual, am not going to know accurately about, say, Scottish county’s. Information online seems rather disagreeable too. A best guess is to take various maps of the county’s from different years, and compare positions.

Towns have also added a problem. Penarth, and Sully were added as towns. Yet an event in Sully is addressed as Penarth. This is apparently, correct. So now, there’s the problem that there is a lot of towns that are part of other towns. As far as I am aware, Sully is a rather small area and cannot really be regarded as a town.

As people add events, hopefully this will filter out which towns can be discarded. All I have learned from this is that Yahoo’s directory could be a bit sharper.

Whenever I meet people, I am generally asked one of two things – “How’s the sites going?” or “What do you do?” If I reply with a rather general “I develop and run websites” then usually I will be told of some friend of theirs who wants a site developed. Ok, but then when going into details, it turns out to be that they want a design, NOT a development.

People seem to confuse the two. Someone who sits there and writes the code to make the site functional is not often the one who also designs the theme. I can write with PHP and MySQL as if it was a native language – I can work well with Javascript, and if there’s something I can’t do with it, a solution is a search away. Heck, I could find my way around Flash/Actionscript in hardly any time. Each new problem is a lesson. But when it comes to design, I am still very, VERY basic.

Part of this is that designing a site doesn’t interest me much at all. Coding the functionality – the solution to a problem, does. currently, if someone asks me to design a site, I get the most horrible sinking feeling. Even though I’d like to help, I can’t. Again, the problem is time. In order to do what most people want with a site, I need to sit and learn how to edit images properly. This takes time, and is something I am seriously not interested in. I am essentially, not an artist, and have very little, if, at all, inspiration design-wise.

The two problem areas I have with design, are inspiration, and effective image editing. I know enough of CSS and XHTML to create the layout of a site. I don’t know enough of image editing to create the most effective images. My experience goes as far as creating a few gradients for a phpBB2 theme, and a few buttons. The theme itself went very well, with people commenting that, for a phpBB2 theme, it was very good. But this is still the basics – a gradient just involves picking two colours, and drawing a line in the direction you want the colours to interpolate between each other.

Here comes the situation. I will NEED to design soon. The previously mentioned almost released site needs to actually have a design. To develop it, I just pulled in an old PHP framework which included a custom-coded templating system, and a theme – a rather poor theme at that. It is irritatingly blue. My worry with picking a good design is not focussed just on the colours and images – I am taking each page into consideration. Should the main menu be on the side of the page? Could it still be clear as a row underneath the site’s header? How will I re-arrange the event details page? Should the Google map be where it is? Is it obvious that there is time and cost information underneath the map?

The Usability Test

People always say that, no matter how obvious something may seem, you should always get someone who hasn’t seen it before, to use it, and observe what happens. I sat someone in front of the site, who has only recently started using Facebook – as far as computers go, she can send and recieve emails, and write letters, but cannot set up an email account, or install an Office suite.

She navigated the site somewhat ok. She found the map showing the location of each event. She did not realise that you could see more information, and even the link to the event, by clicking a map marker. This, despite there being a “key” above the map showing what the markers are for. There are other ways to get to the events – in particular, a listing of all events, with a search filter above. This also failed. The events were listed, but she did not know to actually click an event to see more information. The mouse cursor changed to a hand symbol, to indicate that it was clickable, but she didn’t try it. The same thing happened in a seperate documents section – she also did not click to read a document, as “there was no link”. To be fair, there actually was no link – the table rows were clickable, and would navigate to the actual content. To me, it was obvious there was a link, as soon as the cursor would change, and that a list of documents would indicate that the actual content is elsewhere (and would be pointed to by the list).

Considering that the site is aimed for “average user”, this is a problem that needs to be sorted along with the new theme.

A few years ago, almost all my efforts went into web development. This was before I took an interest in games development. Now I seem to have gone back to the web development full time, because of a small issue in life – that bunch of numbers representing how much you are worth, with a currency sign in front of it.

I can’t say i’m all dissapointed though – although PHP/MySQL gets very monotonous, Javascript, at the moment, hasn’t. It just takes me a lot longer to develop something because I am still very new to Javascript. I have come across some stupid errors recently, mostly due to lack of experience with how Javascript likes to be used. Typically these have been found and fixed, with a few facepalm moments, but it’s all good. The result, is a far more satisfying user experience.

The quickest example I can give – if someone wishes to add content to the site – they give the content a title. As they type (After three characters have been entered), javascript performs a search for existing content title’s, and returns the results inside a div element. The user straight away (well, almost – of course there is the delay in sending the request to the server, and getting a response) gets a dynamic check to see whether they are about to add a potentially existing item. They can also click the items retrieved as they type, to see if it really is exactly the same. This is by no means unique or new, but just something that could actually save a lot of dissapointed users adding something which they thought didn’t exist. This is by no means the pinnacle of what I’ve been doing – just an example.

I have now clocked up about 6 years of web development, and only recently I have taken Javascript very seriously. Before, it felt like a hacky, badly supported, and often disabled feature. But seeing other, very strong websites such as Facebook, and most things Google use it, and inspire others to use it rather heavily, it seems about time I used it too. Where sensible, that is.

Now for the Ranter

This makes me want to give a comment about university. In the three years, we very roughly touched on Ajax techniques, and in that time, very poorly. I doubt many students there will ever use it again, and out of those who would, would have a confused understanding of it – if they have it working in one browser, they will wonder why it isn’t working in another, and then they would give up there, other than a few remaining students who don’t need someone to hold their hand. After three years of university, someone should not have to feel like a complete beginner to a “modern technology”, which the university prides itself on keeping up with. By all means, don’t hold our hands, but give us the time to sit and use it. This is how it typically went:

- University touches a subject briefly, and gives an assignment on it.

- Student attempts assignment, along with 5 other assignments of other subjects.

- Student doesn’t learn anything, other than a couple functions or concepts, because he/she is too rushed for time, which each assignment is competing heavily for.

- Student hands in something rather poor, and gets a grade making out that they just didn’t get it, when in reality, the student likely just didn’t have the time to sit down and experiment with it.

You start learning programming by seeing how others have coded something. You then play around with it, or code your own version. If you are unsure of anything, you look it up, and understand it. You run into problems, some which leave your mind blank, some which leave you staring at the code, and values during each breakpoint, but you find it eventually. You eventually think up other ways of doing the task, but more efficiently. You get a nice feeling when a tough task works out, and then you carry on with another. You don’t learn by seeing a quick powerpoint slide with horrible coding standards, and then tackling a small task, whilst stressed about 5 other, potentially bigger tasks, not giving you the time to really find out WHY something isn’t working, or is done how it is. Essentially, in a few weeks of using Javascript, I have learnt at least 5 times what we “learnt” during university, on the same subject.

This was going to be injected in somewhere, but during year 2 (where we had to create a site using Flash, and MySQL) – despite doing all that was asked, I had 50%. Another student, who avoided Flash altogether and just used php/MySQL, ended up with a distinction. When trying to get a reason why, the lecturer kept putting it off week after week (during summer), before finally saying “it’s too late now”. If the lecturer can actually tell the difference between a Flash application, and a PHP/MySQL site, then he must have been so fed up of marking, that he just scribbled a number on the front. So, despite working far harder and longer than the other student, and making sure I would get the full marks for a subject I was entirely interested in, I got the “you didn’t try” mark. The other student got the “well done, it mostly works!” mark, despite not even trying.

Don’t waste your money on University.

A lot of people say it all the time – make sure you backup important data! Yet so many people, although they agree, keep putting it off week after week. Me included, until now (hopefully).

The server stores a good few folders full of useful data – by no means “important” to other people, but it would be a pain to have to gather it all again. It also stores a good selection of media files ripped from any CD’s we bought over the years, all with the correct ID3 tags, filenames, and whatnot. It also, obviously serves a few web projects, repositories, etc, which would be a disaster if they were lost, along with MySQL databases.

To back them up, I could just enter a few commands, and job’s done. Or I could do what a programmer would do, and write something to automate this process. In this case, a nice little PHP-CLI script which:

- Mounts the second hard drive

- Copies over the specified directories (termed, “filesystem” backups in case of this script)

- Copies over any service configurations (termed “configs” in case of this script)

- Runs mysqldump on each specified database

- Logs the entire process to /var/log/{script_name}/{date}.log

- Unmounts the second hard drive

To determine what it needs to backup, and where to place it, is configured in a file stored in /etc/{script_name_not_yet_decided}.conf. You just specify as many of each type of backup you wish (filesystem, config, and MySQL), and it will array these, and run through them.

Of course, there are endless tools out there which will do this, but this was a case of wanting a solution that does only exactly what I want, how I want, without any extras, command tricks, and whatnot. I also didn’t want to spend an afternoon learning Perl, or bash script, so I just stuck with a language I know like the back of my hand.

It has been a while since work has been done on the game projects, but by no means is this by a lack of interest. Money is the problem.

Actually, the title is misleading – development has still been going strong – just, not on the games. I have been working on a website – I will not post details until it is released, of what its purpose is. It should be good – hopefully. Oh, and it is not in the slightest related to games.

It is taking a while to develop because I want to cover as many of the bases as I can. Other websites already cover it’s topic, but, in my opinion, not very well. From previous experience with web development, it is a horrible chore to replace functionality once the site is live. It is also very annoying releasing something with various corners cut short. In all, I want the user to have the best experience for the given topic. I want it to be free for the user to use, and I want it to be something they will come back to time and time again.

One purpose of this post is to go into detail with one function available to the site’s admin. Recently, the test server was exploited. A phishing scam was uploaded to the webserver, but luckily, it seems no one was a victim of it, other than the compromised user on the system losing all his files, which were recently backed up nicely. The point is though, the exploit was made possible due to some serious lack of consideration of the security involved. If the server is a castle surrounded with lava, then I was the one who build a concrete bridge and left the front gate open. SSH is now restricted heavily, and user’s passwords which should have been strict, are now strict.

That leaves just the file/directory permissions of the web server. I wrote a script which:

-Lists the contents of the entire webroot, their permissions, and checksums
-Compares this list against a ’snapshot’ of the directory tree stored in the database
-Lists both the file system copy, and the database copy side-by-side, flagging them green if ok, red if:
–md5 checksum of the file does not match
–permissions are not correct
–file/directory from file system could not be found in the database copy

The user (admin!) is given the option to accept the changes to the file after he has checked it’s contents. If the admin has not added, or modified anything, then everything should turn up green. If not, then the script will know about it, and thus, so will the admin. The script can also try to correct the permissions, but unless the files/directories are owned by the apache user, then this is unlikely to work. This is also likely to be the case, as apache simply needs to read the files – it does not need to write to them. The only exception should be a directory which apache has write permissions, for handling file uploads. In this case, the script will let the admin flag it as a writable directory, so that he is not bombarded with hundreds of red entries when the script finds and compares the files in that directory.

To top it all off, a summary is provided after the entire comparison if finished. Saves the admin from scrolling through for signs of red.

By no means a comprehensive security ’solution’, but it provides a nice peace of mind accessble whenever the admin logs in to manage the website.

Screenshot of the filesystem security script

During a get together weekend, KnightEdit saw the biggest joint contribution we have done so far. We both decided to keep out progress to ourselves for a week, and suprise each other. And suprise each other we did!

Terry had been hard at work on a deferred shading system, complete with normal mapping. The results were incredible. After about a year of being stuck with mundane, boring university work and being told we are stupid for thinking we can succeed (ironic for a place that advertises it’s desire to ensure students do succeed), we now have a lighting system at least as good as many examples we saw throughout the previous year, wondering when we will have the same. Terry did a very good job of making the deferred shader very easy to use, making it far easier than I expected to give the map editor such a boost.

The editor has a few new features, most importantly the ability to actually save and load maps! It doesn’t just load a map, but also loads the data on all selected objects when the map was saved, thus, if the user saves and closes the editor, then when the map is reloaded, any selections will still be highlighted.

KnightEdit's first light

The most impressive change is the implementation of lighting. Light objects can be added, given colours, positioned, and can also have their intensity adjusted. Whether you want a small, dim light in the corner of a room, or a super-bright halogen, it should be possible. Being WYSIWYG, the lighting does not need some “compile” option before you see the results of tweaking – it is done as you go along.

KnightEdit deferred with three lights

Adding a light is as easy as adding any object. It will be placed in the same location as the currently selected object (or wherever the last coordinate of the mouse was if selection mode is off). The properties panel on the right loads up the object’s properies.

It could be improved with a colour picker – it is suprisingly tedious to adjust the colours at the moment, if you want to do it quickly.

The screenshots so far do not do the lighting system justice. For one, none of the textures are normal mapped. I think you can guess what the next screenshots will include.

This is a bit of a sudden entry considering that, as a reader who has only followed this blog, the last bit of “progress” was a post about porting to XNA.

Well, in the months of absence, things have been going on. Mainly planning the future, and what will happen especially in these first few critical months.

A number of websites are being planned and made – one of which is considerably close to being ready. It will require hours of data inputting to beef it up, but it’s “quick content” – and certainly useful for it’s purpose. I will give more details when it is actually released, but for now, it really should become both a useful resource, and nice little revenue stream.

The meat of the money would come from the games – I hope. A lot of effort is going into them, and I have been tasked with making the Map Editor. After all, I spent quite some time using Valve Hammer Editor, FinalAlert/FinalSun, Duke 3D Build, RA2’s Map Editor, and a bit of UTEditor. Nothing much came of any work with Valve Hammer, but it inspired me the most. I liked it’s entity setups and naming scheme, and that is one key area of KnightEdit. The texturing controls will probably be very similar to VHE too.

So, why the name KnightEdit? For now, it is temporary, but if we don’t get around to renaming it, I guess the name will have to do. The name comes from the type of game we are making a ‘clone’ of – KnightLore, from the Spectrum, and Solstice. There are more games like this, but those two are the first that come to mind. The game will probably be more like KnightLore, than the others too – not graphically though.

The game is rather simple compared to most games out there today, but we chose this as a first game for a number of reasons – mainly because, it is something to give us an initial taste of writing games, and gives a good basis for general tools for creating bigger, more complex games.

Our biggest challenge is not the coding, but time itself – coding takes time. Time is money. Therefore, to gain more time, we need more money, and to get more money, we need more time to produce something amazing to make more money. To top it all off, the results from uni come soon, which could make a huge difference to time. The first reaction from parents would be to go and get a “full time job like everyone else”. That, will absolutely kill the game development. Time is running out, things need to be released, and money needs to be made, so we can continue our passion, and also make more money than we would in a “full time job”. Even if we are working longer hours than in a full time job – is a job really “work”, if it is also a passion? It is still a task, and takes effort, but it is a nice kind of work, not a “ugh, it’s Monday again” kind.

Anyhow, at this stage, the map editor can add, move, delete objects, handle different kinds of objects, and apply different textures to objects, and parameters to objects. It just cannot save a map…yet . Entities are very close to being able to trigger each other.

KnightEdit Screenshot