Coding an Entity Manager

[WillSmith190]

Sorry if I'm asking a lot here, but it's disappointing that there is no full Source engine reference that I can find, just "copy this code here" tutorials. There are also very few places to ask for help, and this seems to be the most active one I can find.

Intro:

I'm creating a game that allows players to build with user-defined blocks. There can be upwards of thousands of these blocks, so in order to stay under the entity limit, I'm coding a single entity that handles all the blocks in the server. I am currently in very early development of this manager, so far all I have is a client and server entity, the client one handling rendering, and the server one handling building. When a client places one of these blocks, it tells the server, which in turn notifies all the clients so that they can render them. Unfortunately, I don't really know how to achieve this. There are also a few other networking-related things I need help with.

Issue #1:

First of all, I need to know how to solve the problem mentioned above, about notifying all the clients that a block has been placed. All blocks are quads, but they can be different sizes, defined by a Vector. There are future plans for block types (doors, stairs, etc.) which will be defined by an integer (e.g. 0 == NORMAL_BLOCK, 1 == DOOR_BLOCK, 2 == STAIR_BLOCK). I also need to send the texture. To keep bandwidth low, instead of sending the texture string, I can send an integer that corresponds to a texture on the client's side (e.g. 0 == wood001, 1 == concrete 001, etc.).

Basically, how do I send 2 integers and a Vector to all the clients? Do I use game events? I tried this, but I couldn't seem to find a way to allow the client class to receive the message, which makes me think game events are for the server only. User messages seem to only be for HUD updates, but could I use them in this way, too?

Issue #2:

When a client connects to a server, they need to receive all the blocks that are currently placed in the server. How do I send all this data effectively? This could be a lot of data. I don't think it would be a good idea to send a massive packet of all the data, but rather to send multiple packets split into 2048 bytes. How does the engine populate the client with all the entities? I could also use this method.

Additional Information

Thanks to anyone that can solve one of my issues, and a big thanks to you if you can solve both. Code samples would be nice, because there is a significant lack of Source engine documentation, and a lot of times the concept isn't the problem, rather doing it correctly in the engine. However, I've been programming C++ for about 3 1/2 years now, so I don't need a step by step explanation of pointers.

[stoopdapoop]

Well, you're right in assuming that events aren't the way to go, they can get out of sync and they're not nearly as fancy as the regular entity networking system.

The Networking_Entities page on the VDC should help you out a lot.

You want to create a sendtable of arrays for each element of your blocks, and just look them up by index, I think this will be the best/simplest method.

for an existing example of sending arrays you can check out player_resource.cpp and player_resource.h on the server, and the corresponding c_player_resource.cpp and c_player_resource.h on the client.

The network encoder and decoder offered by source is pretty nice, if you set up your variable encoding properly, then it'll automatically keep everything as small as possible, only transmit data that's changed, and only update when needed. Also, it'll automatically update the clients when they connect to the server, and will split up packets for you. But this is all explained in the VDC article.

[Sandern]

This might interest you, you could consider contacting him. I believe he batched all blocks into a single mesh.

I also did something similar by recreating a dungeon keeper clone (currently up to a grid size of 170x170 tiles/blocks). Each block is represented by a non networked entity (i.e. not using sendtables) and manually synced using user messages. However Source Engine can only render about 2048 entities max at the same time, so this approach might not be interesting for you (depending on what you want). The user messages might also not be the best choice in case you want to sync all blocks at once, because there is a limit to the number of messages you can send at the same time (the game will bug out with a "reliable user message overflow").

[zombie@computer]

Just something to think about: if you do plan on using a grid(-like system) to place your blocks, you can greatly reduce network traffic and thus block limits.

imagine you use a 2024*2024*512 grid. Then you can suffice sending ONE integer instead of the three floats making up a vector (saves 67% of data, not counting overhead/compression).
when using a grid, you can use a single byte for scale in each dimension (assuming no player is going to build something with a block covering > 10% of the playing field in any dimension). Bam, another 75% of data saved.
if you enumerate block types using a char (DONT USE INT!), a single block only utilises 64 bits (32 bits for grid placement, 3*8bits for scale and 1*8bits for blocktype). Alternatively you can put these bits into any other variable if there is room. Add another 16 bits for textures and/or other properties (owner? Door.isopen?), then that makes 80 bits per block. Compare that to a single vector (96bits uncompressed) and you know why playing with bits can really pay off. This way, using standard entity networking, you can transmit around 200 brushes per update.

[WillSmith190]

Hey, thanks a lot for the help!

stoopdapoop:
I had read that article a very long time ago, and thought I remember everything from it. I guess not! I'll re-read it as soon as I'm done typing this. I love when things are done automatically

Sandern:
Although it's a little disappointing my idea has already been used, at least I know it's possible. If I really have problems, I'll see if I can get into contact with him. However, YouTube isn't a very reliable means of contact.

zombie@computer
That scale thing is something I never thought of! I'll definitely be using that. And don't worry, I was going to use an unsigned char for the block types, not an int (what would I do with 65535 values for an unsigned short?)!

Thanks for all the help guys, hopefully this is enough to get me off my feet with this thing.

Update:

Ok, I have an additional question. What is the most data I can send in my send table? The way I'm doing this now is that I have split all the required data into arrays that will be sent over the network in the send table. I can't seem to get it to work though, because it seems sending 8192 (a number far from my expected max blocks) bools is to much for the engine, and it crashes on startup. How can I optimize this? Am I doing this correctly?

MAX_BLOCKS is 8192

Code: Select all

CNetworkArray(bool, m_aBlockValid, MAX_BLOCKS);

Code: Select all

IMPLEMENT_SERVERCLASS_ST(CInfoBlockManager, DT_InfoBlockManager)
   SendPropArray3(SENDINFO_ARRAY3(m_aBlockValid), SendPropBool(SENDINFO_ARRAY(m_aBlockValid))),
END_SEND_TABLE()

Code: Select all

IMPLEMENT_CLIENTCLASS_DT( C_InfoBlockManager, DT_InfoBlockManager, CInfoBlockManager )
   RecvPropArray3(RECVINFO_ARRAY(m_aBlockValid), RecvPropInt(RECVINFO(m_aBlockValid[0]))),
END_RECV_TABLE()

Sorry if I'm misunderstanding this entire thing, I'm not a network programmer

[stoopdapoop]

Code: Select all

SendPropArray3(SENDINFO_ARRAY3(m_aBlockValid), SendPropBool(SENDINFO_ARRAY(m_aBlockValid), 1, SPROP_UNSIGNED))

would probably help, booleans take up a fully byte in C++ (up to 4 bytes in a struct if not packed properly) and I bet they send the whole thing unless you tell it to only send the first bit.

[WillSmith190]

Code: Select all

SendPropArray3(SENDINFO_ARRAY3(m_aBlockValid), SendPropBool(SENDINFO_ARRAY(m_aBlockValid), 1, SPROP_UNSIGNED))

would probably help, booleans take up a fully byte in C++ (up to 4 bytes in a struct if not packed properly) and I bet they send the whole thing unless you tell it to only send the first bit.

This is weird. IntelliSense says that will work, but compiling it says it doesn't take 5 args. I didn't think I was passing 5 args...

[stoopdapoop]

weird, try SendPropInt instead of SendPropBool maybe? That's how valve passes bools, and how I've done it in the past. Looks like your receive table is already expecting an int anyway.

Edit: Fixed my ebonics

[WillSmith190]

weird, try SendPropInt instead of SendPropBool maybe? That's how valve passes bools, and how I've done it in the past. Looks like you're already receive table is expecting an int anyway.

Oh, that being in the recv table was a mistake

I'll try out using SendPropInt, but I can't test it because my client class doesn't seem to want to link with the server class (breakpoints in Spawn and DrawModel aren't hit...). I know I'm missing something silly, when I figure it out I'll let you know how its working.

Edit:

Ok, I got it to work, I was just missing the UpdateTransmitState() function. It seems that some of the data is being sent, but not all of it. First of all, all the bools are true, even though in Spawn(), I set them to false. I will try moving this, because maybe it doesn't work in Spawn()? Also, only the x value of the position and scale is being sent. I think I'm doing it wrong, I just tried recycling a technique I would use for passing data to OpenGL and DirectX shaders.

I have these structs, which should just be raw data (unless I'm missing some compiler setting):

Code: Select all

typedef struct
{
   int x, y, z;
} IntVector3D;

typedef struct
{
   unsigned char x, y, z;
} CharVector3D;

I use them to hold the position and scale, instead of using a Vector (because I don't need floats. There is an IntVector4D, but I don't need the 4th dimension). I send them like this:

Code: Select all

SendPropArray3(SENDINFO_ARRAY3(m_aBlockPositions), SendPropInt(SENDINFO_ARRAY(m_aBlockPositions), sizeof(int)*3, 0)),
   SendPropArray3(SENDINFO_ARRAY3(m_aBlockScales), SendPropInt(SENDINFO_ARRAY(m_aBlockScales), 3, SPROP_UNSIGNED)),

The first on being 3 ints, so its sizeof(int)*3, and the second one is 3 unsigned chars, so its just 3 bytes. The receiving end puts the data into the same structs. I'm assuming this is wrong, because it doesn't work.

How do I send a 3D int vector and a 3D char vector over the network? Do I have to write my own encoding and decoding function? I can't send an int[3] or char[3], it doesn't like that

Edit II:

It seems that on the receiving end, the entire array is being filled with the values I'm passing (I'm trying to set block 0 to position 2,3,4 and scale 5,6,7. All the array items are 2,0,0 for position, and 5,0,0 for scale (remember, only the x value is getting through)).

[zombie@computer]

This is how i sent int[]s:

Code: Select all

SendPropArray(SendPropInt(SENDINFO_ARRAY(myint),0),myint),


[WillSmith190]

This is how i sent int[]s:

Code: Select all

SendPropArray(SendPropInt(SENDINFO_ARRAY(myint),0),myint),


I'm sending an array of int[3], and it won't let me send an array of arrays

[zombie@computer]

an array of arrays? why the hell would you do that? Just flatten the array.

[WillSmith190]

an array of arrays? why the hell would you do that? Just flatten the array.

What, you mean make it 384 in size, and just have every 3 items be a coord? That seems so ugly... but if there's not support for an array of arrays or an array of structs, than I guess I have to do this.

EDIT:
Ok, I did that, but now I get spammed with "Entity 44 'info_block_manager' reported ENTITY_CHANGE_NONE but '001' changed." And the 001 can be a 002, 003, 010, and a bunch of other values. Am I overflowing memory somewhere?

Code: Select all

SendPropArray3(SENDINFO_ARRAY3(m_aBlockPositions), SendPropInt(SENDINFO_ARRAY(m_aBlockPositions), (sizeof(int)*3)*3, 0)),
   SendPropArray3(SENDINFO_ARRAY3(m_aBlockScales), SendPropInt(SENDINFO_ARRAY(m_aBlockScales), 3*3, SPROP_UNSIGNED)),
   SendPropArray3(SENDINFO_ARRAY3(m_aBlockTypes), SendPropInt(SENDINFO_ARRAY(m_aBlockTypes), 1, SPROP_UNSIGNED)),
   SendPropArray3(SENDINFO_ARRAY3(m_aBlockTypes), SendPropInt(SENDINFO_ARRAY(m_aBlockTextures), 1, SPROP_UNSIGNED)),
   SendPropArray3(SENDINFO_ARRAY3(m_aBlockValid), SendPropInt(SENDINFO_ARRAY(m_aBlockValid), 1, SPROP_UNSIGNED)),

After a few of those messages the game crashes, any ideas?

[zombie@computer]

you can easily map a multidimensional array to a single dimensional one using pointers if you want to, so i don't see the problem. Why make things harder for yourself? Also you realize there's a 2kb update max per entity?