An electrocorticography-based speech decoder for neural speech prostheses – Nature.com

Neural speech prostheses primarily based on mind–laptop interface (BCI) programs and machine studying applied sciences are elementary instruments for serving to restore speech loss as a consequence of neurological deficits. Regardless of the progress, advances in speech decoding are nonetheless hampered by the restricted timeframe of information obtainable for coaching customized fashions and the inherent variability in speech manufacturing. Now, writing in Nature Machine Intelligence, Adeen Flinker and colleagues introduce an electrocorticography (ECoG)-to-speech framework that overcomes information shortage and enhances natural-sound speech synthesis.

The strategy is predicated on a particular acoustic house that may synthesize speech utilizing differentiable digital sign processing and an acoustic pre-training that’s blind to the neural information and allow to decode sensible vocalizations with restricted information. The coaching course of includes semi-supervised studying that originally makes use of solely speech indicators, adopted by supervised studying primarily based on ground-truth spectrograms (utilizing measures of spectrogram distinction and short-time goal intelligibility, STOI). The group used varied deep studying architectures (together with 3D ResNet, 3D Swin Transformer and LSTM) that reply on totally different causality instructions for the ECoG decoder and educated the fashions utilizing a portion of the information and validated them on the remaining set. Strong decoding efficiency have been achieved throughout contributors with each high-density and low-density electrode sampling on the cortex and excessive accuracy was demonstrated throughout totally different fashions, with causal decoding with convolutional architectures displaying superior efficiency.

This can be a preview of subscription content material, entry through your establishment

if (window.dataLayer) {
window.dataLayer.push({
content material: { article: { relevantArticlesCount: 0 }}
})
}

Entry choices

/* type specs begin */
type{show:none!vital}.LiveAreaSection-193358632 *{align-content:stretch;align-items:stretch;align-self:auto;animation-delay:0s;animation-direction:regular;animation-duration:0s;animation-fill-mode:none;animation-iteration-count:1;animation-name:none;animation-play-state:working;animation-timing-function:ease;azimuth:middle;backface-visibility:seen;background-attachment:scroll;background-blend-mode:regular;background-clip:borderBox;background-color:clear;background-image:none;background-origin:paddingBox;background-position:0 0;background-repeat:repeat;background-size:auto auto;block-size:auto;border-block-end-color:currentcolor;border-block-end-style:none;border-block-end-width:medium;border-block-start-color:currentcolor;border-block-start-style:none;border-block-start-width:medium;border-bottom-color:currentcolor;border-bottom-left-radius:0;border-bottom-right-radius:0;border-bottom-style:none;border-bottom-width:medium;border-collapse:separate;border-image-outset:0s;border-image-repeat:stretch;border-image-slice:100%;border-image-source:none;border-image-width:1;border-inline-end-color:currentcolor;border-inline-end-style:none;border-inline-end-width:medium;border-inline-start-color:currentcolor;border-inline-start-style:none;border-inline-start-width:medium;border-left-color:currentcolor;border-left-style:none;border-left-width:medium;border-right-color:currentcolor;border-right-style:none;border-right-width:medium;border-spacing:0;border-top-color:currentcolor;border-top-left-radius:0;border-top-right-radius:0;border-top-style:none;border-top-width:medium;backside:auto;box-decoration-break:slice;box-shadow:none;box-sizing:border-box;break-after:auto;break-before:auto;break-inside:auto;caption-side:high;caret-color:auto;clear:none;clip:auto;clip-path:none;shade:preliminary;column-count:auto;column-fill:stability;column-gap:regular;column-rule-color:currentcolor;column-rule-style:none;column-rule-width:medium;column-span:none;column-width:auto;content material:regular;counter-increment:none;counter-reset:none;cursor:auto;show:inline;empty-cells:present;filter:none;flex-basis:auto;flex-direction:row;flex-grow:0;flex-shrink:1;flex-wrap:nowrap;float:none;font-family:preliminary;font-feature-settings:regular;font-kerning:auto;font-language-override:regular;font-size:medium;font-size-adjust:none;font-stretch:regular;font-style:regular;font-synthesis:weight type;font-variant:regular;font-variant-alternates:regular;font-variant-caps:regular;font-variant-east-asian:regular;font-variant-ligatures:regular;font-variant-numeric:regular;font-variant-position:regular;font-weight:400;grid-auto-columns:auto;grid-auto-flow:row;grid-auto-rows:auto;grid-column-end:auto;grid-column-gap:0;grid-column-start:auto;grid-row-end:auto;grid-row-gap:0;grid-row-start:auto;grid-template-areas:none;grid-template-columns:none;grid-template-rows:none;top:auto;hyphens:guide;image-orientation:0deg;image-rendering:auto;image-resolution:1dppx;ime-mode:auto;inline-size:auto;isolation:auto;justify-content:flexStart;left:auto;letter-spacing:regular;line-break:auto;line-height:regular;list-style-image:none;list-style-position:exterior;list-style-type:disc;margin-block-end:0;margin-block-start:0;margin-bottom:0;margin-inline-end:0;margin-inline-start:0;margin-left:0;margin-right:0;margin-top:0;mask-clip:borderBox;mask-composite:add;mask-image:none;mask-mode:matchSource;mask-origin:borderBox;mask-position:0 0;mask-repeat:repeat;mask-size:auto;mask-type:luminance;max-height:none;max-width:none;min-block-size:0;min-height:0;min-inline-size:0;min-width:0;mix-blend-mode:regular;object-fit:fill;object-position:50% 50%;offset-block-end:auto;offset-block-start:auto;offset-inline-end:auto;offset-inline-start:auto;opacity:1;order:0;orphans:2;outline-color:preliminary;outline-offset:0;outline-style:none;outline-width:medium;overflow:seen;overflow-wrap:regular;overflow-x:seen;overflow-y:seen;padding-block-end:0;padding-block-start:0;padding-bottom:0;padding-inline-end:0;padding-inline-start:0;padding-left:0;padding-right:0;padding-top:0;page-break-after:auto;page-break-before:auto;page-break-inside:auto;perspective:none;perspective-origin:50% 50%;pointer-events:auto;place:static;quotes:preliminary;resize:none;proper:auto;ruby-align:spaceAround;ruby-merge:separate;ruby-position:over;scroll-behavior:auto;scroll-snap-coordinate:none;scroll-snap-destination:0 0;scroll-snap-points-x:none;scroll-snap-points-y:none;scroll-snap-type:none;shape-image-threshold:0;shape-margin:0;shape-outside:none;tab-size:8;table-layout:auto;text-align:preliminary;text-align-last:auto;text-combine-upright:none;text-decoration-color:currentcolor;text-decoration-line:none;text-decoration-style:strong;text-emphasis-color:currentcolor;text-emphasis-position:over proper;text-emphasis-style:none;text-indent:0;text-justify:auto;text-orientation:blended;text-overflow:clip;text-rendering:auto;text-shadow:none;text-transform:none;text-underline-position:auto;high:auto;touch-action:auto;rework:none;transform-box:borderBox;transform-origin:50% 50percent0;transform-style:flat;transition-delay:0s;transition-duration:0s;transition-property:all;transition-timing-function:ease;vertical-align:baseline;visibility:seen;white-space:regular;widows:2;width:auto;will-change:auto;word-break:regular;word-spacing:regular;word-wrap:regular;writing-mode:horizontalTb;z-index:auto;-webkit-appearance:none;-moz-appearance:none;-ms-appearance:none;look:none;margin:0}.LiveAreaSection-193358632{width:100%}.LiveAreaSection-193358632 .login-option-buybox{show:block;width:100%;font-size:17px;line-height:30px;shade:#222;padding-top:30px;font-family:Harding,Palatino,serif}.LiveAreaSection-193358632 .additional-access-options{show:block;font-weight:700;font-size:17px;line-height:30px;shade:#222;font-family:Harding,Palatino,serif}.LiveAreaSection-193358632 .additional-login>li:not(:first-child)::earlier than{rework:translateY(-50%);content material:””;top:1rem;place:absolute;high:50%;left:0;border-left:2px strong #999}.LiveAreaSection-193358632 .additional-login>li:not(:first-child){padding-left:10px}.LiveAreaSection-193358632 .additional-login>li{show:inline-block;place:relative;vertical-align:center;padding-right:10px}.BuyBoxSection-683559780{show:flex;flex-wrap:wrap;flex:1;flex-direction:row-reverse;margin:-30px -15px 0}.BuyBoxSection-683559780 .box-inner{width:100%;top:100%;padding:30px 5px;show:flex;flex-direction:column;justify-content:space-between}.BuyBoxSection-683559780 p{margin:0}.BuyBoxSection-683559780 .readcube-buybox{background-color:#f3f3f3;flex-shrink:1;flex-grow:1;flex-basis:255px;background-clip:content-box;padding:0 15px;margin-top:30px}.BuyBoxSection-683559780 .subscribe-buybox{background-color:#f3f3f3;flex-shrink:1;flex-grow:4;flex-basis:300px;background-clip:content-box;padding:0 15px;margin-top:30px}.BuyBoxSection-683559780 .subscribe-buybox-nature-plus{background-color:#f3f3f3;flex-shrink:1;flex-grow:4;flex-basis:100%;background-clip:content-box;padding:0 15px;margin-top:30px}.BuyBoxSection-683559780 .title-readcube,.BuyBoxSection-683559780 .title-buybox{show:block;margin:0;margin-right:10%;margin-left:10%;font-size:24px;line-height:32px;shade:#222;text-align:middle;font-family:Harding,Palatino,serif}.BuyBoxSection-683559780 .title-asia-buybox{show:block;margin:0;margin-right:5%;margin-left:5%;font-size:24px;line-height:32px;shade:#222;text-align:middle;font-family:Harding,Palatino,serif}.BuyBoxSection-683559780 .asia-link{shade:#069;cursor:pointer;text-decoration:none;font-size:1.05em;font-family:-apple-system,BlinkMacSystemFont,”Segoe UI”,Roboto,Oxygen-Sans,Ubuntu,Cantarell,”Helvetica Neue”,sans-serif;line-height:1.05em6}.BuyBoxSection-683559780 .access-readcube{show:block;margin:0;margin-right:10%;margin-left:10%;font-size:14px;shade:#222;padding-top:10px;text-align:middle;font-family:-apple-system,BlinkMacSystemFont,”Segoe UI”,Roboto,Oxygen-Sans,Ubuntu,Cantarell,”Helvetica Neue”,sans-serif;line-height:20px}.BuyBoxSection-683559780 ul{margin:0}.BuyBoxSection-683559780 .link-usp{show:list-item;margin:0;margin-left:20px;padding-top:6px;list-style-position:inside}.BuyBoxSection-683559780 .link-usp span{font-size:14px;shade:#222;font-family:-apple-system,BlinkMacSystemFont,”Segoe UI”,Roboto,Oxygen-Sans,Ubuntu,Cantarell,”Helvetica Neue”,sans-serif;line-height:20px}.BuyBoxSection-683559780 .access-asia-buybox{show:block;margin:0;margin-right:5%;margin-left:5%;font-size:14px;shade:#222;padding-top:10px;text-align:middle;font-family:-apple-system,BlinkMacSystemFont,”Segoe UI”,Roboto,Oxygen-Sans,Ubuntu,Cantarell,”Helvetica Neue”,sans-serif;line-height:20px}.BuyBoxSection-683559780 .access-buybox{show:block;margin:0;margin-right:10%;margin-left:10%;font-size:14px;shade:#222;opacity:.8px;padding-top:10px;text-align:middle;font-family:-apple-system,BlinkMacSystemFont,”Segoe UI”,Roboto,Oxygen-Sans,Ubuntu,Cantarell,”Helvetica Neue”,sans-serif;line-height:20px}.BuyBoxSection-683559780 .price-buybox{show:block;font-size:30px;shade:#222;font-family:-apple-system,BlinkMacSystemFont,”Segoe UI”,Roboto,Oxygen-Sans,Ubuntu,Cantarell,”Helvetica Neue”,sans-serif;padding-top:30px;text-align:middle}.BuyBoxSection-683559780 .price-buybox-to{show:block;font-size:30px;shade:#222;font-family:-apple-system,BlinkMacSystemFont,”Segoe UI”,Roboto,Oxygen-Sans,Ubuntu,Cantarell,”Helvetica Neue”,sans-serif;text-align:middle}.BuyBoxSection-683559780 .price-info-text{font-size:16px;padding-right:10px;shade:#222;font-family:-apple-system,BlinkMacSystemFont,”Segoe UI”,Roboto,Oxygen-Sans,Ubuntu,Cantarell,”Helvetica Neue”,sans-serif}.BuyBoxSection-683559780 .price-value{font-size:30px;font-family:-apple-system,BlinkMacSystemFont,”Segoe UI”,Roboto,Oxygen-Sans,Ubuntu,Cantarell,”Helvetica Neue”,sans-serif}.BuyBoxSection-683559780 .price-per-period{font-family:-apple-system,BlinkMacSystemFont,”Segoe UI”,Roboto,Oxygen-Sans,Ubuntu,Cantarell,”Helvetica Neue”,sans-serif}.BuyBoxSection-683559780 .price-from{font-size:14px;padding-right:10px;shade:#222;font-family:-apple-system,BlinkMacSystemFont,”Segoe UI”,Roboto,Oxygen-Sans,Ubuntu,Cantarell,”Helvetica Neue”,sans-serif;line-height:20px}.BuyBoxSection-683559780 .issue-buybox{show:block;font-size:13px;text-align:middle;shade:#222;font-family:-apple-system,BlinkMacSystemFont,”Segoe UI”,Roboto,Oxygen-Sans,Ubuntu,Cantarell,”Helvetica Neue”,sans-serif;line-height:19px}.BuyBoxSection-683559780 .no-price-buybox{show:block;font-size:13px;line-height:18px;text-align:middle;padding-right:10%;padding-left:10%;padding-bottom:20px;padding-top:30px;shade:#222;font-family:-apple-system,BlinkMacSystemFont,”Segoe UI”,Roboto,Oxygen-Sans,Ubuntu,Cantarell,”Helvetica Neue”,sans-serif}.BuyBoxSection-683559780 .vat-buybox{show:block;margin-top:5px;margin-right:20%;margin-left:20%;font-size:11px;shade:#222;padding-top:10px;padding-bottom:15px;text-align:middle;font-family:-apple-system,BlinkMacSystemFont,”Segoe UI”,Roboto,Oxygen-Sans,Ubuntu,Cantarell,”Helvetica Neue”,sans-serif;line-height:17px}.BuyBoxSection-683559780 .tax-buybox{show:block;width:100%;shade:#222;padding:20px 16px;text-align:middle;font-family:-apple-system,BlinkMacSystemFont,”Segoe UI”,Roboto,Oxygen-Sans,Ubuntu,Cantarell,”Helvetica Neue”,sans-serif;line-height:NaNpx}.BuyBoxSection-683559780 .button-container{show:flex;padding-right:20px;padding-left:20px;justify-content:middle}.BuyBoxSection-683559780 .button-container>*{flex:1px}.BuyBoxSection-683559780 .button-container>a:hover,.Button-1078489254:hover,.Button-2737859108:hover{text-decoration:none}.BuyBoxSection-683559780 .btn-secondary{background:#fff}.BuyBoxSection-683559780 .button-asia{background:#069;border:1px strong #069;border-radius:0;cursor:pointer;show:block;padding:9px;define:0;text-align:middle;text-decoration:none;min-width:80px;margin-top:75px}.BuyBoxSection-683559780 .button-label-asia,.ButtonLabel-3296148077,.ButtonLabel-1636778223{show:block;shade:#fff;font-size:17px;line-height:20px;font-family:-apple-system,BlinkMacSystemFont,”Segoe UI”,Roboto,Oxygen-Sans,Ubuntu,Cantarell,”Helvetica Neue”,sans-serif;text-align:middle;text-decoration:none;cursor:pointer}.Button-1078489254,.Button-2737859108{background:#069;border:1px strong #069;border-radius:0;cursor:pointer;show:block;padding:9px;define:0;text-align:middle;text-decoration:none;min-width:80px;max-width:320px;margin-top:20px}.Button-1078489254 .btn-secondary-label,.Button-2737859108 .btn-secondary-label{shade:#069}
/* type specs finish */

Purchase this text

Buy on Springer Hyperlink
Immediate entry to full article PDF

Purchase now

Costs could also be topic to native taxes that are calculated throughout checkout

References

Authentic article

Chen, X. et al. A neural speech decoding framework leveraging deep studying and speech synthesis. Nat. Mach. Intell. 6, 467–480 (2024)

Article

Google Scholar

Obtain references

Writer data

Authors and Affiliations

Nature Evaluations Electrical Engineering https://www.nature.com/natrevelectreng/

Silvia Conti

Authors

Silvia Conti

View writer publications

You can too seek for this writer in
PubMed Google Scholar

Corresponding writer

Correspondence to
Silvia Conti.

Rights and permissions

Reprints and permissions

About this text

Cite this text

Conti, S. An electrocorticography-based speech decoder for neural speech prostheses.
Nat Rev Electr Eng (2024). https://doi.org/10.1038/s44287-024-00054-5

Obtain quotation

Printed: 01 Could 2024
DOI: https://doi.org/10.1038/s44287-024-00054-5

Adblock take a look at (Why?)