Future API Sketch

[Gozala]

Me @alanshaw, @vasco-santos and @travis had been talking about revisiting API for the client at the labweek and have got some feedback from the workshop which was validating of the one we were entertaining, I thought I'd sketch it here so we can discuss it further. Before I proceed let me call out some high level goals I have in my mind.

1. Client state should be fully exposed and really minimal. I think it should just be a "delegation store" + "connection config".
2. Client itself should be just a "view" class over the state "model" that simply exposes static functions operating on the "model" as methods as a convenience DSL.
   • E.g. queries over proofs should not be a clients methods (especially private) they should be just static functions over proof store and we could choose to expose or conceal them.
   • I should be able to create various "view" classes simply by wrapping over "model" and not worry that there is some state that may be out of sync.
3. We should have views for relevant entities in the system e.g.
   • AgentView - Convenience wrapper of what I have access to on this agent.
   • AccountView - Agent may have access to arbitrary number of accounts represented via AccountView which should simply be another view over the "proofs store". When you login you get a new AccountView (holding account to agent delegation) if you do want to persist reference to the account across the agent sessions you do agent.add(account).
     • SpaceView - Agent may have access to arbitrary number of spaces represented via SpaceView which should also be views over "proofs store". When you create a space you get a new OwnedSpaceView. If you want to retain access to space across devices and sessions you'd call account.add(space). If you do not want to add space to the account you can add it it e.g. agent.local.add(space) or something like that so we emphasize the fact that space is detached from any account.
       • You could also have account.remove(space) which simply revokes delegation from space to account.
   • UploadView and StoreView will be just be views that expose corresponding upload/* and store/* capabilities of the space. I don't mean subclasses here, simply views over the same model. And space could probably have .upload and and .store fields with the corresponding views.
4. View methods should simply query proofs in the model to find relevant delegations and invoke them if they have or fail right away if they don't.

[Gozala]

This is roughly the interface I had in mind when talking about it

interface ConnecionModel {
  /**
   * Local keypair use to sign messages send over the connection.
   */
  readonly issuer: Ucanto.Signer
  /**
   * DID of the destination.
   */
  readonly audience: Ucanto.Principal
  /**
   * Wire transport encoding.
   */
  readonly codec: Ucanto.Transport.OutboundCodec
  /**
   * Ucanto Transport channel.
   */
  readonly channel: Ucanto.Transport.Channel
}


interface Model {
  /**
   * Connection to the (remote) agent a.k.a service.
   */
  connection: ConnecionModel
  /**
   * Set of delegations that can be used to invoke capabilities over `connection`.
   */
  authorization: Store<Ucanto.Delegation>
}

interface AgentView {
  model: Model

  accounts: AgentAccounts
  login(email: string): Promise<AccountView>
}


interface AgentAccounts extends Iterable<AccountView> {
  get(email: string): AccountView
  add(account: AccountView): Promise<AccountView>
  remove(account: AccountView): Promise<void>

}

interface AccountView {
  spaces: Spaces
  plan: AccountPlan
}

interface AccountPlan {
  get(): PlanView
}

interface PlanView {
  subscriptions: Subscriptions
}

interface Subscriptions extends Iterable<Subscription> {
  get(id: string): Subscription
  add(subscription: Subscription): Promise<Subscription>
  remove(subscription: Subscription): Promise<void>
}

interface Subscription {
  consumer: SpaceDID
}

interface Spaces extends Iterable<SpaceView> {
  get(name: string): SpaceView
  add(space: SpaceView): Promise<SpaceView>
  remove(space: SpaceView): Promise<void>
}

interface SpaceView {
  upload: UploadView
  store: StoreView

  createFileUploader(): FileUploader
  createDirectoryUploader(): DirectoryUploader
  createArchiveUploader(): ArchiveUploader
}

export interface Uploader {
  /**
   * Writes contents of the upload to the space without adding it to the upload
   * list.
   */
  store(space?: SpaceView): Promise<Upload>
  /**
   * Writes content of the upload to the space and adds it to the upload list.
   */
  upload(space?: SpaceView): Promise<Upload>
}

interface FileUploader extends Uploader {
}

interface DirectoryUploader extends Uploader {
}

interface ArchiveUploader extends Uploader {
}

interface Upload {
  root: Link
  shards: CARLink[]
}

interface UploadView {
  add(upload: Upload): Promise<Upload>
  remove(upload: Upload): Promise<Unit>
  list(): Promise<Upload[]>
}

interface StoreView {
  add(shard: Archive): Promise<ArchiveWriter>
  remove(shard: Archive): Promise<Unit>
  list(): Promise<Archive[]>
}

interface Archive {
  link: Link

  size: number
}

interface ArchiveWriter {
  write(bytes: Uint8Array): Promise<void>
}

[alanshaw]

This looks great, really nice to be able to design something like this now we have our domain model figured out. I realise just a sketch but here's some notes on missing/wanted things so we don't forget:

• Spaces have a human readable name that should probably be exposed in the SpaceView?
• Subscriptions can have multiple consumers
• Need to expose Filecoin deal information for a given shard, we have filecoin/info capability but we can probably have a nicer API that models the aggregation chain, using receipts to fetch information for each stage
• How do we cater for the case when we already have a shard and do not need to transfer it again?
• Listing store/upload items should be paginated
• We have space usage reports (usage/report) that can only be retrieved for spaces your account has provisioned
• If we can we should be careful with the directory uploader API - either encouraging or enforcing a stable ordering for files - we've (I've) been bitten already with uploads of the same files but provided in different order

[hannahhoward]

@alanshaw I've heard about the client refactor, and I'd love to hear from you what you see as the goal of this refactor, in qualitative terms.

• are we trying to make it easier for people to use our client library?
• are we trying to make it easier for development to happen on our client library? (i.e. more maintainable perhaps)
• are we trying to reduce complexity in our client library?

Essentially what makes worth it to spend 2 weeks at minimum to implement all this plus the pain of getting folks to upgrade? I think it would super useful also for @Peeja as someone with deep frontend experience to have strong input on this.

I will say for myself by far one of the hardest parts of working in the w3up repo is the abundance of packages and the fact that a typical operational function will cross potentially multiple different packages. (and on the server side, into the w3infra repo and back)

Also, would it help to defer this for a bit until @hakierka and @Peeja have a few weeks experience working on the code and /or helping clients so they can have more informed input?

[alanshaw]

I'd love to hear from you what you see as the goal of this refactor, in qualitative terms.

• Better DX. The client is split between 3 packages access-client, upload-client, both consumed by w3up-client.
  • This would consolidate into a single package which makes it easier to understand, work with, open PRs against.
  • Removes a ton of boilerplate re-exporting types and values in access-client, upload-client from w3up-client.
  • Removes possibility of accidentally releasing a breaking change in a minor/patch access-client and upload-client that adversely effects w3up-client.
• Better UX. The API grew organically as we solidified our domain model with different teams working on different aspects of the client. We now have a better idea of our domain objects and the proposal above is an idea to model the client around them.
  • Stateful client has proven to be not good UX. For example, the client has the concept of a "current space" which all interactions work against. We've seen in multiple situations users have had trouble understanding what space they're uploading data to. It's also easy to upload to the wrong space and hard to know what the client should do when importing a new space (should current space be set to it?). It's also a bit of a security concern when creating delegations because, again, it's easy to create a delegation for a space you didn't intend.
  • Where we have forgotten to re-export types/values from w3up-client we're occasionally had to ask users to install access-client and upload-client to access them - we don't really want this as the installed versions might not be compatible when managed by the user and not by the w3up-client.
• Smaller delegations. The client currently "overshares" delegations and includes anything that could possibly be relevant rather than the most succinct proof chain. This is more of an issue as time progresses.
• Mostly complete. There is a PR open where most of the work has already been done. It needs to be polished and it needs the blob protocol ported to it. feat: unified client #1306

I've likely missed a few things here...

FYI we have a planning ticket open for this also: storacha/project-tracking#2

My ask in discord was not for this to be worked on by @hakierka or @Peeja just yet, I just wondered, given the proposed API and the in-flight PR, if, (in their opinion!) the proposed API might be better than the current, given they are new to the project, just learning the ropes and experiencing the pains other users might also be experiencing. I also think their opinion is valuable given their expertise. This would provide us with some more validation and hence would give us more reason to work on it.