Ubuhanga bwo gukora imibonano mpuzabitsina si abahanga gusa cyangwa abafasha mu kuvuga bigana abantu. Inyuma y'ibyo byose, hari umusozi - rimwe na rimwe uba ari inyanja - w'amakuru. Kandi mu by'ukuri, kubika ayo makuru? Aho niho ibintu bikunze kuba akajagari. Waba uvuga uburyo bwo kumenya amashusho cyangwa utoza abahanga mu ndimi, ibisabwa mu kubika amakuru kuri ubu buryo bwo gukora imibonano mpuzabitsina bishobora kugorana vuba niba utabitekerejeho neza. Reka turebe impamvu kubika ari ikintu gikomeye, ni amahitamo kihari, n'uburyo ushobora guhuza ikiguzi, umuvuduko, n'uburyo bwo kugabanya ibiciro nta gucika intege.
Ingingo ushobora gukunda gusoma nyuma y'iyi:
🔗 Ubumenyi bw'amakuru n'ubwenge bw'ubukorano: Ahazaza h'udushya
Gusuzuma uburyo ubuhanga bwo gukora imibonano mpuzabitsina n'ubumenyi bw'amakuru bitera udushya dugezweho.
🔗 Ubwenge bw'amazi y'ubukorano: Ahazaza h'ubuhanga bwo gukora imibonano mpuzabitsina n'amakuru akoreshwa mu buryo bwa “decentrally”
Irebere ku makuru ya AI yagabanyijwe hamwe n'udushya tukiri kugaragara.
🔗 Gucunga amakuru ku bikoresho bya AI ukwiye kureba
Ingamba z'ingenzi zo kunoza uburyo bwo kubika no gukoresha neza amakuru ya AI.
🔗 Ibikoresho byiza bya AI ku basesenguzi b'amakuru: Kongera ubushobozi bwo gufata ibyemezo mu isesengura
Ibikoresho by'ubuhanga mu bya siyansi (AI) bifasha mu gusesengura amakuru no gufata ibyemezo.
None se… Ni iki gituma kubika amakuru ya AI biba byiza? ✅
Si "terabytes nyinshi" gusa. Ububiko nyabwo bworoshye gukoresha AI bugamije gukoreshwa, kwizerwa, no kwihuta bihagije haba mu myitozo ndetse no mu mirimo ijyanye n'ibizamini.
Ibintu bike by'ingenzi bikwiye kwitabwaho:
-
Guhindagurika : Kuva kuri GBs ujya kuri PBs udahinduye imiterere yawe.
-
Imikorere : Gutinda cyane bizatera GPU kwicwa n'inzara; ntibibabarira ibibazo.
-
Ubuhinduzi : Amafoto, gukoporora, guhindura - kuko igerageza ritagenda neza, kandi n'abantu barabikora.
-
Kunoza ikiguzi : Urwego rukwiye, igihe gikwiye; bitabaye ibyo, umushinga w'itegeko uhita winjira nk'igenzura ry'imisoro.
-
Ahantu ho kubara : Shyira ububiko iruhande rwa GPU/TPU cyangwa urebe uko amakuru atangwa ahagarara.
Bitabaye ibyo, ni nko kugerageza gukoresha Ferrari ikoresheje lisansi yo guca ubwatsi - mu by'ukuri iragenda, ariko ntimara igihe kirekire.
Imbonerahamwe yo kugereranya: Amahitamo asanzwe yo kubika amakuru kuri AI
| Ubwoko bw'ububiko | Uburyo bwiza bwo gukora | Pariki y'umupira w'amaguru y'igiciro | Impamvu Ikora (cyangwa Idakora) |
|---|---|---|---|
| Ububiko bw'ibintu mu bicu | Amasosiyete mato n'ay'ibigo bito biciriritse | $$ (ihindagurika) | Yoroshye, iramba, ikwiriye cyane ibiyaga by'amakuru; witondere amafaranga yo gusohoka + gusaba gutsindwa. |
| NAS yo mu nyubako | Imiryango minini ifite amakipe y'ikoranabuhanga | $$$$ | Gutinda guteganyijwe, kugenzura byuzuye; capex y'imbere + ikiguzi cy'imikorere ihoraho. |
| Igicu cy'uruhererekane | Imiterere ikomeye y'amategeko | $$$ | Ihuza umuvuduko wo mu gace runaka n'igicu cyoroshye; imiterere yongera umutwe. |
| Urutonde rwa All-Flash | Abashakashatsi bashishikajwe cyane n'ibyo bakora | $$$$$ | IOPS/throughput yihuta cyane; ariko TCO ntabwo ari urwenya. |
| Sisitemu z'amadosiye zikwirakwizwa | Abahanga mu bya AI / amatsinda ya HPC | $$–$$$ | I/O ijyanye n'igihe ku gipimo gikomeye (Lustre, Spectrum Scale); umutwaro wa ops ni nyawo. |
Impamvu ibyifuzo by'amakuru ya AI biri kwiyongera cyane 🚀
Ubuhanga bwo gukora imibonano mpuzabitsina (AI) si ukwibika amafoto gusa. Ni ikintu kibi cyane.
-
Amahugurwa : ILSVRC yonyine ya ImageNet ipakiye amashusho afite amazina ya ~1.2M, na sosiyete yihariye ya domaine irenga ibyo [1].
-
Guhindura : Buri guhindura - ibirango, ibice, kongeramo - bitanga ikindi "kuri".
-
Ibikoresho byo kureba amashusho : Ishusho imbonankubone, telemetri, uburyo bwo kubyaza amakuru… ni umuriro uhoraho.
-
Imiterere idateye neza : Inyandiko, videwo, amajwi, ububiko - binini cyane kurusha imbonerahamwe za SQL zisukuye.
Ni buffet isanzwe igushimisha, kandi icyitegererezo gihora kigaruka ku meza.
Cloud vs On-Premises: Ikiganiro kidashira 🌩️🏢
Cloud isa n'ishishikaje: hafi nta mupaka, isi yose, wishyura uko ugenda. Kugeza igihe fagitire yawe igaragaza amafaranga yo gusohoka - kandi mu buryo butunguranye ibiciro byawe "bihendutse" byo kubika bihangana n'amafaranga yo gukoresha mu kubara [2].
Ku rundi ruhande, on-prem itanga ubuyobozi n'imikorere ikomeye, ariko kandi wishyura ibikoresho, ingufu, gukonjesha, n'ibikoresho byo kwita ku bana.
Amakipe menshi ahora mu mwanya utari mwiza: rya hybrid . Bika amakuru ashyushye, y’ibanga, kandi afite ubushobozi bwo gushyiramo amakuru menshi hafi ya GPU, hanyuma ubike andi yose mu byiciro bya cloud.
Ibiciro byo kubika ibintu byiyongera cyane 💸
Ubushobozi ni urwego rwo hejuru gusa. Amafaranga yihishe ararundanya:
-
Ihindagurika ry'amakuru : Kopi hagati y'uturere, kohereza amakuru mu bicu bitandukanye, ndetse no gusohoka kw'abakoresha [2].
-
Gusubiramo : Nyuma ya 3-2-1 (kopi eshatu, ibitangazamakuru bibiri, kimwe kitari aho) birya umwanya ariko bikiza umunsi [3].
-
Ingufu n'ubukonje : Niba ari icyuma cyawe cyo gukonjesha, ni ikibazo cyawe cy'ubushyuhe.
-
Guhuza hagati y'ibice by'urubura : Imiterere ihendutse ikunze gusobanura ko umuvuduko wo kugarura urubura.
Umutekano no kubahiriza amategeko: Uburyo bwo guhosha amasezerano mu buryo butuje 🔒
Amabwiriza ashobora kugena aho byte ziba. Muri gahunda ya GDPR yo mu Bwongereza , kwimura amakuru bwite mu Bwongereza bisaba inzira zemewe n'amategeko zo kohereza amakuru (SCCs, IDTAs, cyangwa amategeko ahagije). Ubuhinduzi: igishushanyo mbonera cy'ububiko bwawe kigomba "kumenya" imiterere y'ahantu [5].
Ibintu by'ibanze byo guteka kuva ku munsi wa mbere:
-
Guhisha amakuru - haba kuruhuka no mu ngendo.
-
Uburyo bwo kubona amakuru make cyane + inzira zo kugenzura.
-
Gusiba uburyo bwo kurinda nko kudahinduka cyangwa gufunga ibintu.
Imbogamizi mu mikorere: Gutinda ni byo byica bucece ⚡
GPU ntizikunda gutegereza. Iyo ububiko butinze, ziba ari ibyuma bishyushya bigezweho. Ibikoresho nka NVIDIA GPUDirect Storage bigabanya ubushobozi bwa CPU, bihita bivana amakuru kuri NVMe bijya kuri GPU memory - icyo amahugurwa menshi yifuza [4].
Ibikoresho bisanzwe:
-
NVMe yose-flash ku bikoresho bishyushye byo kwitoza.
-
Sisitemu za dosiye zihurirana (Lustre, Spectrum Scale) kugira ngo habeho uburyo bwinshi bwo gukoresha node.
-
Async loaders hamwe na sharding + prefetch kugira ngo GPU zidakora.
Ingamba Zifatika zo Gucunga Ububiko bwa AI 🛠️
-
Ingano : Ibice bishyushye kuri NVMe / SSD; ububiko buhora buhinduka ibice by'ikintu cyangwa ibice bikonje.
-
Dedup + delta : Bika imirongo y'ibanze inshuro imwe, komeza diffs + manifest gusa.
-
Amategeko agenga imikorere : Hindura ibisohoka bishaje byikora kandi birangira [2].
-
Gukomera kwa 3-2-1 : Buri gihe shyira kopi nyinshi, mu bitangazamakuru bitandukanye, hamwe n'imwe yonyine [3].
-
Ibikoresho : Uburyo bwo gukoresha umurongo, igihe cyo gutinda kuri p95/p99, uburyo bwo gusoma bwananiranye, uburyo bwo kuva mu mwanya w'akazi.
Agasanduku k'imashini (kakozwe ariko gasanzwe) 📚
Itsinda ry’abareba ibintu ritangirana n’ububiko bw’ibintu mu bicu bwa TB bigera kuri 20. Nyuma, batangira gukora igerageza ryo gukusanya amakuru mu turere twose. Amafaranga yabo ni menshi cyane - atari ayo mu bubiko ubwabwo, ahubwo ni ayo mu muvuduko w’abantu basohoka . Bimura ibice bishyushye kuri NVMe hafi y’itsinda rya GPU, babika kopi yemewe mu bubiko bw’ibintu (hamwe n’amategeko agenga ubuzima), kandi bagashyiramo ingero bakeneye gusa. Umusaruro: GPU zirimo akazi kenshi, fagitire ni nke, kandi isuku y’amakuru irarushaho kuba nziza.
Igenamigambi ry'ubushobozi bw'inyuma y'ibahasha 🧮
Uburyo bwo gusesengura:
Ubushobozi ≈ (Itsinda ry'amakuru risanzwe) × (Igipimo cyo gukoporora) + (Amakuru yateguwe mbere / yongerewe) + (Aho kugenzura + Inyandiko) + (Umupaka w'umutekano ~ 15–30%)
Hanyuma urebe neza niba ureba imikorere ya bandwidth. Niba per-node loaders zikeneye ~ 2–4 GB/s zihoraho, ureba NVMe cyangwa parallel FS ku nzira zishyushye, hamwe n'ububiko bw'ibintu nk'ukuri kw'ibanze.
Ntabwo ari iby'umwanya gusa 📊
Iyo abantu bavuze ibisabwa mu kubika amakuru y’ubukorano bwa gihanga (AI) , batekereza kuri terabytes cyangwa petabytes. Ariko amayeri nyayo ni uburinganire: ikiguzi ugereranyije n’imikorere, ubworoherane ugereranyije n’iyubahirizwa ry’amategeko, guhanga udushya ugereranyije n’ihindagurika. Amakuru y’ubukorano bwa gihanga ntabwo ari kugabanuka vuba aha. Amakipe ashyira ububiko mu buryo bw’icyitegererezo mu gihe gito yirinda kurohama mu bishanga by’amakuru - kandi barangiza nabo bakitoza vuba.
Amareferensi
[1] Russakovsky n'abandi. ImageNet Large Scale Visual Recognition Challenge (IJCV) — ingano y'amakuru n'ikibazo. Link
[2] AWS — Amazon S3 Ibiciro n'ibiciro (kohereza amakuru, gusohoka, urwego rw'ubuzima). Link
[3] CISA — Inama ku mategeko yo kubika amakuru 3-2-1. Link
[4] NVIDIA Docs — Incamake y'ububiko bwa GPUDirect. Link
[5] ICO — Amategeko ya GDPR y'Ubwongereza ku kohereza amakuru mpuzamahanga. Link