C API for ArrayBuffer/TypedArray

[martijnthe]

As per Gilles'/@barbibulle 's request on the mailing list:

Do you have any expectations of the interface? Which abilities are urgent in your application? Those feedbacks will help us design the API.

what I'd expect from the public interface is to be able to:
1/ get to the underlying memory buffer of an ArrayBuffer and TypedArray, using something equivalent to what's in ecma-arraybuffer-object.h and:

• ecma_arraybuffer_get_buffer (ecma_object_t *obj_p);
• ecma_arraybuffer_get_length (ecma_object_t *obj_p);
• ecma_typedarray_get_buffer (ecma_object_t *typedarray_p);
• ecma_typedarray_get_length (ecma_object_t *typedarray_p);
• ecma_typedarray_get_element_size_shift (ecma_object_t *typedarray_p);

2/ Ability to query whether a jerry_value_t is an instance of ArrayBuffer or is a TypedArray. Something equivalent to:

• ecma_is_arraybuffer (ecma_value_t val);
• ecma_is_typedarray (ecma_value_t target);

3/ Maybe also a convenience function for creating an ArrayBuffer with a single call (probably not necessary for TypedArrays). Something equivalent to:

• ecma_arraybuffer_new_object (ecma_length_t lengh);

4/ It would be nice to also have DataView implemented, since Uint32Array is always with the > platform's endianness, it makes it hard to parse external data structures that are big endian when running on little-endian platforms.

When interfacing with native APIs, including filesystems, network interfaces, etc, ArrayBuffers are very convenient, and the ability to easily exchange buffers with JS code is extremely useful.

[jiangzidong]

Rough plan for C API:

ArrayBuffer

Though JS spec did provide buffer-access ability for ArrayBuffer, here, in native API, we provide them.

• bool jerry_value_is_arraybuffer(jerry_value_t);
• jerry_value_t jerry_create_arraybuffer(jerry_length_t);
• jerry_length_t jerry_arraybuffer_set_buffer(jerry_value_t val, uint8_t *buf_p, jerry_length_t buf_size); It will copy the content in buf_p into the buffer of arraybuffer. return the number of actually copied value.
• jerry_length_t jerry_arraybuffer_get_buffer(jerry_value_t val, uint8_t *buf_p, jerry_length_t buf_size); it will copy the content in arraybuffer into the buf_p. return the number of actually copied value.
• jerry_length_t jerry_arraybuffer_get_length(jerry_value_t);

TypedArray

//used to indicate the certain type in jerry_create_typedarray
typedef enum
{
  JERRY_TYPEDARRAY_UINT8,
  JERRY_TYPEDARRAY_INT8,
  ...
} jerry_typedarray_class_t

• bool jerry_value_is_typedarray(jerry_value_t);
• jerry_value_t jerry_create_typedarray(jerry_length_t length, jerry_typedarray_class_t class_name);
• jerry_typedarray_class_t jerry_typedarray_get_type(jerry_value_t);
• jerry_length_t jerry_typedarray_get_length(jerry_value_t);
• jerry_value_t jerry_typedarray_get_buffer(jerry_value_t val, jerry_length_t *byteLength, jerry_length_t *byteOffset); return the arraybuffer object inside the typedarray, and also the byteLength and byteOffset.

What do you think of the above proposal?
@barbibulle @LaszloLango @zherczeg @martijnthe

[LaszloLango]

Sounds good

[barbibulle]

A couple questions:

1/ for jerry_length_t jerry_arraybuffer_set_buffer(uint8_t *buf_p, jerry_length_t buf_size);
and
jerry_length_t jerry_arraybuffer_get_buffer(uint8_t *buf_p, jerry_length_t buf_size);
I think you're missing a jerry_value_t parameter for the ArrayBuffer object

2/ would it be possible to consider providing direct buffer access without copy (assuming that the underlying buffer representation is directly memory-mapped to the actual bytes)? Something like:
uint8_t* jerry_arraybuffer_use_buffer(jerry_value_t arraybuffer);
(this would allow read/write access directly to the underlying buffer memory)

Also, Is there a plan to also support DataView? If so, then something like:
bool jerry_arraybuffer_is_view(jerry_value_t)
might be useful (along with functions to access the buffer of the DataView, just like for jerry_typedarray_get_buffer, etc.)

[martijnthe]

+1 for having no-copy APIs.

To keep consistent terminology, how about:
uint8_t *jerry_arraybuffer_acquire_buffer(jerry_value_t arraybuffer);
void jerry_arraybuffer_release_buffer(jerry_value_t arraybuffer);
The calls would also acquire/release the jerry_value_t to make sure the object is not gc'd for as long as the native side is using the buffer.
It would be the responsibility of the client to make sure to only access the buffer from the same task/thread as the VM runs, to avoid concurrent access.

[zherczeg]

Hm, I would keep the normal acquire/release API for it, and simply provide a getter function.

[jiangzidong]

Sorry for the late reply.
@barbibulle

1/ forget the ArrayBuffer object

Yes, stupid mistake... Thanks

2/ directly access to the buffer

The reason I use copy because I don't want to expose the details in jerry_heap to user.

• 1, If we return the actual start address of the internal buffer (uint8_t *), and user out-of-bounds access the array and change the items for other object , then whole jerry system may fail.
• 2, now the buffer is a whole block of memory in jerry_heap, but maybe in the future, the internal impl upgrades, and we use linked block to represent arraybuffer. Then the returned address is not a simple start of a buffer.
  What do you think @zherczeg @martijnthe

3/ DataView

In my future plan, but not in the "near" future. Sorry that I am busy on company's internal project.

May I know what do you use (or plan to use) jerry for? Is that a public project?
(If it is secret, please just ignore this question :P )

[zherczeg]

@jiangzidong, you are right. However we are aiming for low-end devices, where copying data and allocating temporary buffers could be too much overhead. And since the buffer is static, we likely will not allocate it in segments. Perhaps if we would want to support segmented buffers, we could make it compile time enabled.

Ok I have no definite opinion about this. I wouldn't mind having a direct access to buffer: encrypting, image/video coding would be much simpler. But Zidong has good reasons.

Iterators could also be an option.

[martijnthe]

If we return the actual start address of the internal buffer (uint8_t *), and user out-of-bounds access the array and change the items for other object , then whole jerry system may fail.

IMO it's the responsibility of the user (C programmer) to make sure this does not happen.
There will always be many C programmer errors that can be made that will make the whole VM fail/crash the system. Such is life in C :)

2, now the buffer is a whole block of memory in jerry_heap, but maybe in the future, the internal impl upgrades, and we use linked block to represent arraybuffer. Then the returned address is not a simple start of a buffer.

I thought the whole point of ArrayBuffer etc are created is to be able to represent a fixed-length, contiguous blob of memory in JS a performant way. OTOH, for JerryScript, a smaller memory footprint is probably more important than runtime speed. Segmenting (large) buffers may help make better use of a constrained heap.

I like the idea of having an iterator API to get a pointer + readable size of the current segment and a way to get the next segment. The first implementation pass could assume a single segment, but this way the API leaves the door open to adding segmentation.

[zherczeg]

I think a segmented buffer would hurt performance too much. It is just too complicated. JerryScript had segmented strings before, and they were causing too much trouble.

[niklauslee]

I really need such APIs for ArrayBuffer/TypedArray. Any progress on this?

[LaszloLango]

@galpeter is working on it. He will open a PR soon.