not all attributes are read from file

[wiesehahn]

I have a file which already contains some extra attributes, if we add more not all of them are read by lidR anymore.

In this case we added xyz normals to the file via lasR, when we read the file we can see that normalY and normalZ are not read.
Is this due to some limit on extry attributes?

lasR::info(ans)
Source       : LASF (v1.4)
Size         : 2.76 GB
Extent       : 528000.00 529000.00 5735000.00 5736000.00 (xmin, xmax, ymin, ymax)
Points       : 36.33 millions
Area         : 999998.0 m²
Density      : 36.3 pts/m²
Coord. ref.  : ETRS89 / UTM zone 32N
Schema       :
34 attributes | 76 bytes per points
 Name: flags             | uchar  | Desc: Internal 8-bit mask reserved for lasR core engine
 Name: X                 | int    | Desc: X coordinate
 Name: Y                 | int    | Desc: Y coordinate
 Name: Z                 | int    | Desc: Z coordinate
 Name: Intensity         | ushort | Desc: Pulse return magnitude
 Name: ReturnNumber      | uchar  | Desc: Pulse return number for a given output pulse
 Name: NumberOfReturns   | uchar  | Desc: Total number of returns for a given pulse
 Name: Classification    | uchar  | Desc: The 'class' attributes of a point
 Name: UserData          | uchar  | Desc: Used at the user’s discretion
 Name: PointSourceID     | short  | Desc: Source from which this point originated
 Name: ScanAngle         | float  | Desc: Angle at which the laser point was output
 Name: ScannerChannel    | uchar  | Desc: Channel (scanner head) of a multi-channel system
 Name: gpstime           | double | Desc: Time tag value at which the point was observed
 Name: R                 | ushort | Desc: Red image channel
 Name: G                 | ushort | Desc: Green image channel
 Name: B                 | ushort | Desc: Blue image channel
 Name: NIR               | ushort | Desc: Near infrared channel value
 Name: Amplitude         | ushort | Desc: Echo signal amplitude [dB]
 Name: Pulse width       | ushort | Desc: Full width at half maximum [ns]
 Name: Reflectance       | short  | Desc: Echo signal reflectance [dB]
 Name: Deviation         | ushort | Desc: Pulse shape deviation
 Name: confidence        | uchar  | Desc: confidence values
 Name: Distance          | int    | Desc: Distance
 Name: Group             | uint   | Desc: Group
 Name: Normal            | uint   | Desc: Normal vector 2+15+15 bits
 Name: normalX           | float  | Desc: x component of normal vector
 Name: normalY           | float  | Desc: y component of normal vector
 Name: normalZ           | float  | Desc: z component of normal vector
 Name: EdgeOfFlightline  | bit    | Desc: Set when the point is at the end of a scan
 Name: ScanDirectionFlag | bit    | Desc: Direction in which the scanner mirror was traveling 
 Name: Synthetic         | bit    | Desc: Point created by a technique other than direct observation
 Name: Keypoint          | bit    | Desc: Point is considered to be a model key-point
 Name: Withheld          | bit    | Desc: Point is supposed to be deleted)
 Name: Overlap           | bit    | Desc: If set, point is within an overlap region of 2+ swaths

> las <- lidR::readLAS(ans)                                                                               48s     
> names(las@data)
 [1] "X"                 "Y"                 "Z"                 "gpstime"           "Intensity"        
 [6] "ReturnNumber"      "NumberOfReturns"   "ScanDirectionFlag" "EdgeOfFlightline"  "Classification"   
[11] "ScannerChannel"    "Synthetic_flag"    "Keypoint_flag"     "Withheld_flag"     "Overlap_flag"     
[16] "ScanAngle"         "UserData"          "PointSourceID"     "R"                 "G"                
[21] "B"                 "NIR"               "Amplitude"         "Pulse width"       "Reflectance"      
[26] "Deviation"         "confidence"        "Distance"          "Group"             "Normal"           
[31] "normalX"          

[Jean-Romain]

rlas reads up to 9 extra attributes. This limit exist for 10 years now. So far it has never really been an issue. The reason is because options select expects one character per attribute with digit used for extra attributes (e.g. select = "xyzi23"). This limits to 9.

[bi0m3trics]

I'm just going to throw this out there - in the past when I've needed to create a ton of variables for each point, rather than request/create a change that would cause the .las/.laz files to be astronomical in size, I just write out a .csv file that matches the las@data (excluding the stuff that's in there already) and then if I need it later I amend it as needed.
If needed, it's not too hard to create a custom unique point ID to facilitate the join at a later date.

[Jean-Romain]

@bi0m3trics laz is compressed. las are binary. csv is plain text . The size of csv is far more astronomical.

[bi0m3trics]

The point is there are ways to save the data entirely separate from the existing .las/.laz structure, which can also be compressed, and for my simple example you could/should get 70 to 90% compression on a csv into a zip without modifying the existing file structure of the .las/.laz and it could be compressed/uncompressed from the console or your script.

[wiesehahn]

rlas reads up to 9 extra attributes. This limit exist for 10 years now. So far it has never really been an issue. The reason is because options select expects one character per attribute with digit used for extra attributes (e.g. select = "xyzi23"). This limits to 9.

I had exactly this reason in mind.
Its not a real problem on or side. Files with this many extra attributes are quite rare and actually most of the attributes contain not much information if any. I guess it was just the sensor/software of the providing company which exported these attributes by default.
So in our case we could just add some remove_attribute() stages or once merged to main a remove_attributes()/keep_attributes() stage to the pipeline.