!!array tag for fast read-in ?

--===============3259634166151061318==
Content-Type: multipart/signed; protocol="application/pkcs7-signature";
	micalg=sha-256; boundary="------------ms090809090607030209060802"

--------------ms090809090607030209060802
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: quoted-printable

We consider YAML as a candidate for combining metadata with 256x256 or 10=
24x1204 images
(actually physical detector counts, mostly small integers, which makes AS=
CII storage
quite efficient).
Unfortunately, yamlcpp and libyaml are both too slow for our purpose;
they perform 1-2 orders of magnitude slower than JSON or purpose-written =
ASCII parsers.

In plain YAML, our image data reside in a sequence of sequences, like
- image:
   - [ 0 3 2 0 8 ... ]
   - [ 1 0 7 3 2 ... ]
   - ...
This is inefficient because the parser has no knowledge about this peculi=
ar
data structure, and therefore spends lots of time with unnecessary action=
s.
Is this right?

What do you think of the following concept:

Define a tag !!array, to be used as follows:
- image: !!array 2 uint8 256 256 |
     0 3 2 0 8 ...
     1 0 7 3 2 ...

The number immediately after the tag (here 2) indicates the rank D;
the next word (here uint8) is the data type;
the next D numbers (here 256 and 256) indicate the size of the multidimen=
ional array.
This then is followed by the image data themselves.

In our read-in routine, upon encountering the tag '!!array',
we would switch from the parser of libyaml to a parser of our own
that takes full advantage of our knowledge of what entries must follow an=
 '!!array' tag.

Does this make sense?
Is it consistent with letter and intent of the YAML specs?

Or what else should we do to reconcile YAML with our need for speed?

- Joachim


--------------ms090809090607030209060802
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature

MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCC
D9YwggTVMIIDvaADAgECAghQTsb1PRG0ZDANBgkqhkiG9w0BAQsFADBxMQswCQYDVQQGEwJE
RTEcMBoGA1UEChMTRGV1dHNjaGUgVGVsZWtvbSBBRzEfMB0GA1UECxMWVC1UZWxlU2VjIFRy
dXN0IENlbnRlcjEjMCEGA1UEAxMaRGV1dHNjaGUgVGVsZWtvbSBSb290IENBIDIwHhcNMTQw
NzIyMTIwODI2WhcNMTkwNzA5MjM1OTAwWjBaMQswCQYDVQQGEwJERTETMBEGA1UEChMKREZO
LVZlcmVpbjEQMA4GA1UECxMHREZOLVBLSTEkMCIGA1UEAxMbREZOLVZlcmVpbiBQQ0EgR2xv
YmFsIC0gRzAxMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA6ZvDZ4X5Da71jVTD
llA1PWLpbkztlNcAW5UidNQg6zSP1uzAMQQLmYHiphTSUqAoI4SLdIkEXlvg4njBeMsWyyg1
OXstkEXQ7aAAeny/Sg4bAMOG6VwrMRF7DPOCJEOMHDiLamgAmu7cT3ir0sYTm3at7t4m6O8B
r3QPwQmi9mvOvdPNFDBP9eXjpMhim4IaAycwDQJlYE3t0QkjKpY1WCfTdsZxtpAdxO3/NYZ9
bzOz2w/FEcKKg6GUXUFr2NIQ9Uz9ylGs2b3vkoO72uuLFlZWQ8/h1RM9ph8nMM1JVNvJEzSa
cXXFbOqnC5j5IZ0nrz6jOTlIaoytyZn7wxLyvQIDAQABo4IBhjCCAYIwDgYDVR0PAQH/BAQD
AgEGMB0GA1UdDgQWBBRJt8bP6D0ff+pEexMp9/EKcD7eZDAfBgNVHSMEGDAWgBQxw3kbuvVT
1xfgiXotF2wKsyudMzASBgNVHRMBAf8ECDAGAQH/AgECMGIGA1UdIARbMFkwEQYPKwYBBAGB
rSGCLAEBBAICMBEGDysGAQQBga0hgiwBAQQDADARBg8rBgEEAYGtIYIsAQEEAwEwDwYNKwYB
BAGBrSGCLAEBBDANBgsrBgEEAYGtIYIsHjA+BgNVHR8ENzA1MDOgMaAvhi1odHRwOi8vcGtp
MDMzNi50ZWxlc2VjLmRlL3JsL0RUX1JPT1RfQ0FfMi5jcmwweAYIKwYBBQUHAQEEbDBqMCwG
CCsGAQUFBzABhiBodHRwOi8vb2NzcDAzMzYudGVsZXNlYy5kZS9vY3NwcjA6BggrBgEFBQcw
AoYuaHR0cDovL3BraTAzMzYudGVsZXNlYy5kZS9jcnQvRFRfUk9PVF9DQV8yLmNlcjANBgkq
hkiG9w0BAQsFAAOCAQEAYyAo/ZwhhnK+OUZZOTIlvKkBmw3Myn1BnIZtCm4ssxNZdbEzkhth
Jxb/w7LVNYL7hCoBSb1mu2YvssIGXW4/buMBWlvKQ2NclbbhMacf1QdfTeZlgk4y+cN8ekvN
TVx07iHydQLsUj7SyWrTkCNuSWc1vn9NVqTszC/Pt6GXqHI+ybxA1lqkCD3WvILDt7cyjrEs
jmpttzUCGc/1OURYY6ckABCwu/xOr24vOLulV0k/2G5QbyyXltwdRpplic+uzPLl2Z9Tsz6h
L5Kp2AvGhB8Exuse6J99tXulAvEkxSRjETTMWpMgKnmIOiVCkKllO3yG0xIVIyn8LNrMOVtU
FzCCBV4wggRGoAMCAQICBxekJJt5EN8wDQYJKoZIhvcNAQELBQAwWjELMAkGA1UEBhMCREUx
EzARBgNVBAoTCkRGTi1WZXJlaW4xEDAOBgNVBAsTB0RGTi1QS0kxJDAiBgNVBAMTG0RGTi1W
ZXJlaW4gUENBIEdsb2JhbCAtIEcwMTAeFw0xNDA1MjcxNDU0MDNaFw0xOTA3MDkyMzU5MDBa
MIGDMQswCQYDVQQGEwJERTEnMCUGA1UEChMeRm9yc2NodW5nc3plbnRydW0gSnVlbGljaCBH
bWJIMSowKAYDVQQDEyFGWkogQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkgLSBHMDIxHzAdBgkq
hkiG9w0BCQEWEGNhQGZ6LWp1ZWxpY2guZGUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQCjJkgFDQw08spMsBreCaehMMeNnMjetOXbtn+OPtrLl1gNACj6sI2FPnheE0qaHVjT
QntJWNaztNYPEbPPxg3mgrW08LanC11rshG+N3nFfEIaa5mauxVt15jvJb2uCCDbFArciFST
iQuPlirxQLBJXtmPWSZtftmyZL99Jc9Cx53YT3KXKK7AKVcOyHae41WnRdnu6CGK0qyq6jEV
Y/l+rDi5ENF3S7RJsQzBww9x2qR79uBiW6d/yYBo3RJZKNTuaKEfIQ8BzJlR0n+yglcSYw/E
Gk/pQlpvCJH4fRFp5ifPewZtkLWg//IgaJBA+YvWI2Av8pWesytikQTQHywRAgMBAAGjggH9
MIIB+TASBgNVHRMBAf8ECDAGAQH/AgEBMA4GA1UdDwEB/wQEAwIBBjARBgNVHSAECjAIMAYG
BFUdIAAwHQYDVR0OBBYEFCC9cYmlySozLxHgH+WUJlYPEQIlMB8GA1UdIwQYMBaAFEm3xs/o
PR9/6kR7Eyn38QpwPt5kMBsGA1UdEQQUMBKBEGNhQGZ6LWp1ZWxpY2guZGUwgYgGA1UdHwSB
gDB+MD2gO6A5hjdodHRwOi8vY2RwMS5wY2EuZGZuLmRlL2dsb2JhbC1yb290LWNhL3B1Yi9j
cmwvY2FjcmwuY3JsMD2gO6A5hjdodHRwOi8vY2RwMi5wY2EuZGZuLmRlL2dsb2JhbC1yb290
LWNhL3B1Yi9jcmwvY2FjcmwuY3JsMIHXBggrBgEFBQcBAQSByjCBxzAzBggrBgEFBQcwAYYn
aHR0cDovL29jc3AucGNhLmRmbi5kZS9PQ1NQLVNlcnZlci9PQ1NQMEcGCCsGAQUFBzAChjto
dHRwOi8vY2RwMS5wY2EuZGZuLmRlL2dsb2JhbC1yb290LWNhL3B1Yi9jYWNlcnQvY2FjZXJ0
LmNydDBHBggrBgEFBQcwAoY7aHR0cDovL2NkcDIucGNhLmRmbi5kZS9nbG9iYWwtcm9vdC1j
YS9wdWIvY2FjZXJ0L2NhY2VydC5jcnQwDQYJKoZIhvcNAQELBQADggEBAGcfjE2bMmdDPtAO
fYpK/o0pjafjSbtpFoWyn+Tjm27098SbSk068ud4neDTfyDkqVKUJodGFDX4qBmr2YqzsD00
pbnQEQKxToRRffl3dz67z4QlhLtVxQdwW2PPvP7FSABflXdEMFLGa4YcWMpcnxRBOG9ahC05
BZqP/F5YUUrfjgGKM2S/U2NjP3O0po/SNv+OmL9/LJSkUwSO4HmT9j8ptAfYcYkJJtnG/gyb
aa5Hss890XJbmLZ9J3E/IS2uQCuQTn15v/sIaNCUPfAo92oFhXOsnhbqFhe7vbm2Eenzuj3s
cPb6NxEZphscTof8O+D6dV1u+RskW4Etm+15Jl0wggWXMIIEf6ADAgECAgcb5j8DN1oGMA0G
CSqGSIb3DQEBCwUAMIGDMQswCQYDVQQGEwJERTEnMCUGA1UEChMeRm9yc2NodW5nc3plbnRy
dW0gSnVlbGljaCBHbWJIMSowKAYDVQQDEyFGWkogQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkg
LSBHMDIxHzAdBgkqhkiG9w0BCQEWEGNhQGZ6LWp1ZWxpY2guZGUwHhcNMTYwODMxMTEzNzIz
WhcNMTkwNzA5MjM1OTAwWjBiMQswCQYDVQQGEwJERTEnMCUGA1UECgweRm9yc2NodW5nc3pl
bnRydW0gSnVlbGljaCBHbWJIMREwDwYDVQQLDAhKQ05TLU1MWjEXMBUGA1UEAwwOSm9hY2hp
bSBXdXR0a2UwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCujdAoD6ZAsut2nbwV
1sNmEibm3PaEXQfm/AXWZfrPeZf4RbDpWDIy3ychxmjiJPhxuwHNjTTt8EKLplTrUAPb1V5J
Rj4BydrsMmDNT5T8kZ+ROHxjgOo/vUprRQv3T+fsEf6Q/DJL/jBNR2j8+LKh0q37uAduGKxP
x8rBS2fj4bsdtCbnyjYnSiLcigoEAMBrciD2qaUsdJEiYYov7DfBn/EaqJ5vRLMvVj+dC/Pn
tmzZBAPBfQC9OPVN01Kt2kZL6I8XceTecrQiINmAfvFXDGlQ/wR8xewM300tCDcpuSwI7nAF
CA6Goq78IHP3gjBaoThw0TsBQAdWZa7fvkpFAgMBAAGjggIuMIICKjBABgNVHSAEOTA3MBEG
DysGAQQBga0hgiwBAQQDBTARBg8rBgEEAYGtIYIsAgEEAwEwDwYNKwYBBAGBrSGCLAEBBDAJ
BgNVHRMEAjAAMA4GA1UdDwEB/wQEAwIF4DAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUH
AwQwHQYDVR0OBBYEFNV68ySrMFz7kK7JB3iilKgWQFe4MB8GA1UdIwQYMBaAFCC9cYmlySoz
LxHgH+WUJlYPEQIlMCEGA1UdEQQaMBiBFmoud3V0dGtlQGZ6LWp1ZWxpY2guZGUwewYDVR0f
BHQwcjA3oDWgM4YxaHR0cDovL2NkcDEucGNhLmRmbi5kZS9memotY2EvcHViL2NybC9nX2Nh
Y3JsLmNybDA3oDWgM4YxaHR0cDovL2NkcDIucGNhLmRmbi5kZS9memotY2EvcHViL2NybC9n
X2NhY3JsLmNybDCBywYIKwYBBQUHAQEEgb4wgbswMwYIKwYBBQUHMAGGJ2h0dHA6Ly9vY3Nw
LnBjYS5kZm4uZGUvT0NTUC1TZXJ2ZXIvT0NTUDBBBggrBgEFBQcwAoY1aHR0cDovL2NkcDEu
cGNhLmRmbi5kZS9memotY2EvcHViL2NhY2VydC9nX2NhY2VydC5jcnQwQQYIKwYBBQUHMAKG
NWh0dHA6Ly9jZHAyLnBjYS5kZm4uZGUvZnpqLWNhL3B1Yi9jYWNlcnQvZ19jYWNlcnQuY3J0
MA0GCSqGSIb3DQEBCwUAA4IBAQBj15Ec3J/W0uQx/k+j+zptyOmtFJ39ws6ssETPMdHyIlPY
53108k3ITRT6CoP7fQ76U2URCNcYgS/KEGIIhBFnlZXYRxcD43jecw5sPeMH5rPBZuQ+64xO
lL+msRR6hX+UDw+32wwjwGb1deeMFTTNwUikQvxy20PONreQqd57wCIYtHrqj+jddLbQhg4d
6luI1ZekaIuMrAQiEUgTuowp6TY/oPASFT3c1BGYnOXsaoNKxHU9V9dsiFm4cOk+nHeu+07B
JoPi2LY4ztscs0+ZINIDpGa9FBDHaSJ08sEP5PP9Io3TDYeunOsFyFv5Gy4ppWVFqUeuEHBl
rzq00BvSMYID3jCCA9oCAQEwgY8wgYMxCzAJBgNVBAYTAkRFMScwJQYDVQQKEx5Gb3JzY2h1
bmdzemVudHJ1bSBKdWVsaWNoIEdtYkgxKjAoBgNVBAMTIUZaSiBDZXJ0aWZpY2F0aW9uIEF1
dGhvcml0eSAtIEcwMjEfMB0GCSqGSIb3DQEJARYQY2FAZnotanVlbGljaC5kZQIHG+Y/Azda
BjANBglghkgBZQMEAgEFAKCCAh8wGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG
9w0BCQUxDxcNMTgwNTA0MTQ1NDA5WjAvBgkqhkiG9w0BCQQxIgQgmL+Ss0hwz/gWfpFVMlGK
M/1iaw8VMciKGi3UiuKslOkwbAYJKoZIhvcNAQkPMV8wXTALBglghkgBZQMEASowCwYJYIZI
AWUDBAECMAoGCCqGSIb3DQMHMA4GCCqGSIb3DQMCAgIAgDANBggqhkiG9w0DAgIBQDAHBgUr
DgMCBzANBggqhkiG9w0DAgIBKDCBoAYJKwYBBAGCNxAEMYGSMIGPMIGDMQswCQYDVQQGEwJE
RTEnMCUGA1UEChMeRm9yc2NodW5nc3plbnRydW0gSnVlbGljaCBHbWJIMSowKAYDVQQDEyFG
WkogQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkgLSBHMDIxHzAdBgkqhkiG9w0BCQEWEGNhQGZ6
LWp1ZWxpY2guZGUCBxvmPwM3WgYwgaIGCyqGSIb3DQEJEAILMYGSoIGPMIGDMQswCQYDVQQG
EwJERTEnMCUGA1UEChMeRm9yc2NodW5nc3plbnRydW0gSnVlbGljaCBHbWJIMSowKAYDVQQD
EyFGWkogQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkgLSBHMDIxHzAdBgkqhkiG9w0BCQEWEGNh
QGZ6LWp1ZWxpY2guZGUCBxvmPwM3WgYwDQYJKoZIhvcNAQEBBQAEggEArC6mITGIVcEY5Pqy
ddWXvbRxK//NWcc5oyUzWtdLBSCu+iKW8Mo/pyZnvXsLNi8BW0gpaRGhsN7mZesFb45d9DZb
aB2Gdp5T/JhzWD9JzlfBwktvvVkZwxwRCHSdSLeUlV1UJkA3kguS/gFAlqZSTIClTbQuSGvM
1Kg3D6Y8jQMBfe7y2U28I4DwirIB92lxxoqvRRiaZvQ0YMMCoMRMStSVFDatU2gD+Xdv4he/
kihEndL3nZXl99+OVy5Z3xN2gu2l3K2ftMlpgyhoIcJAd9fXz5riBnzE1KKnCp6TEXUAyjjX
8nYAnCdY6YY6VzuzxbADSx134ln3FFfqPZ4x9gAAAAAAAA==
--------------ms090809090607030209060802--


--===============3259634166151061318==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
--===============3259634166151061318==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Yaml-core mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/yaml-core

--===============3259634166151061318==--