Thank you, Derek for the answers, but I have another question:

My photon display settings are 640x480@60Hz with 16 bpp (RGB555 format), photon is not running. I'm using devg-radeon 
driver on AGP ATI Radeon 9250 Pro.

After GF initializing and attaching to main display layer, I'm started main display layer format enumeration via 
gf_layer_query() and got:

Main layer index=0

Layer format N0
   format=00001310
   caps=80000004
   alpha_valid_flags=00000000
   alpha_combinations=00000000
   order_caps=00000001
   chromakey_caps=00000000
   src_max_height=4096
   src_max_width=4096
   src_max_viewport_height=480
   src_max_viewport_width=640
   dst_max_height=480
   dst_max_width=640
   dst_min_height=480
   dst_min_width=640
   max_scaleup_x=1
   max_scaleup_y=1
   max_scaledown_x=1
   max_scaledown_y=1
   output_mask=00000000
   vcap_mask=001C2ED8

Layer format N1
   format=00001310
   caps=80000004
   alpha_valid_flags=00000000
   alpha_combinations=00000000
   order_caps=00000001
   chromakey_caps=00000000
   src_max_height=4096
   src_max_width=4096
   src_max_viewport_height=480
   src_max_viewport_width=640
   dst_max_height=480
   dst_max_width=640
   dst_min_height=480
   dst_min_width=640
   max_scaleup_x=1
   max_scaleup_y=1
   max_scaledown_x=1
   max_scaledown_y=1
   output_mask=00000000
   vcap_mask=001C2ED8

End of layer formats list

Format 00001310 is a RGB555 and it is ok with it. All information fileds of gf_layer_info_t structure settings are set 
correctly, except for vcap_mask - it has bogus value.

Why the same layer's pixel format is enumerated twice ? Where are all other formats ?

After this experiment I set 640x480@60Hz using 32 bpp (RGBA8888) through gf_display_set_mode(), after layer reattaching 
it reports the same ! It has RGB565 formats only.

I changed photon default desktop video mode to 640x480@60Hz 32 bpp. After photon shutdown my GF application reports that
 main layer index has: 

Main layer index=0

Layer format N0
   format=00001420
   caps=80000004
   alpha_valid_flags=00000000
   alpha_combinations=00000000
   order_caps=00000001
   chromakey_caps=00000000
   src_max_height=4096
   src_max_width=4096
   src_max_viewport_height=480
   src_max_viewport_width=640
   dst_max_height=480
   dst_max_width=640
   dst_min_height=480
   dst_min_width=640
   max_scaleup_x=1
   max_scaleup_y=1
   max_scaledown_x=1
   max_scaledown_y=1
   output_mask=00000000
   vcap_mask=001C2ED8

Layer format N1
   format=00001420
   caps=80000004
   alpha_valid_flags=00000000
   alpha_combinations=00000000
   order_caps=00000001
   chromakey_caps=00000000
   src_max_height=4096
   src_max_width=4096
   src_max_viewport_height=480
   src_max_viewport_width=640
   dst_max_height=480
   dst_max_width=640
   dst_min_height=480
   dst_min_width=640
   max_scaleup_x=1
   max_scaleup_y=1
   max_scaledown_x=1
   max_scaledown_y=1
   output_mask=00000000
   vcap_mask=001C2ED8

End of layer formats list

I set 640x480@60Hz 8bpp, 15bpp, 16bpp, 32bpp using gf_display_set_mode() function but gf_layer_query() reports only two 
the same available formats: format=00001420, which is RGBA8888.