Skip to content

[Feature Request] Support blocksparse mask like topk, topp #11

Description

@smallscientist1
  • block_mask
AttentionEngine(block_size=BLOCK_SIZE)(
             q, # [batch, seqlenq, head, dimqk]
             k, # [batch, seqlenkv, head, dimqk]
             v, # [batch, seqlenkv, head, dimv]
             block_mask, # [batch, seqlenq//BLOCK_SIZE, seqlenkv//BLOCK_SIZE]
)
  • block_indices
AttentionEngine(block_size=BLOCK_SIZE)(
             q, # [batch, seqlenq, head, dimqk]
             k, # [batch, seqlenkv, head, dimqk]
             v, # [batch, seqlenkv, head, dimv]
             block_indices, # [batch, seqlenq//BLOCK_SIZE, head, MAX_BLOCKS]
             selected_block_num, # [batch, head]
)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions