Debian Codegraph @

lintcmd/runner/runner.go

raw
1 // Package runner implements a go/analysis runner. It makes heavy use
2 // of on-disk caching to reduce overall memory usage and to speed up
3 // repeat runs.
4 //
5 // # Public API
6 //
7 // A Runner maps a list of analyzers and package patterns to a list of
8 // results. Results provide access to diagnostics, directives, errors
9 // encountered, and information about packages. Results explicitly do
10 // not contain ASTs or type information. All position information is
11 // returned in the form of token.Position, not token.Pos. All work
12 // that requires access to the loaded representation of a package has
13 // to occur inside analyzers.
14 //
15 // # Planning and execution
16 //
17 // Analyzing packages is split into two phases: planning and
18 // execution.
19 //
20 // During planning, a directed acyclic graph of package dependencies
21 // is computed. We materialize the full graph so that we can execute
22 // the graph from the bottom up, without keeping unnecessary data in
23 // memory during a DFS and with simplified parallel execution.
24 //
25 // During execution, leaf nodes (nodes with no outstanding
26 // dependencies) get executed in parallel, bounded by a semaphore
27 // sized according to the number of CPUs. Conceptually, this happens
28 // in a loop, processing new leaf nodes as they appear, until no more
29 // nodes are left. In the actual implementation, nodes know their
30 // dependents, and the last dependency of a node to be processed is
31 // responsible for scheduling its dependent.
32 //
33 // The graph is rooted at a synthetic root node. Upon execution of the
34 // root node, the algorithm terminates.
35 //
36 // Analyzing a package repeats the same planning + execution steps,
37 // but this time on a graph of analyzers for the package. Parallel
38 // execution of individual analyzers is bounded by the same semaphore
39 // as executing packages.
40 //
41 // # Parallelism
42 //
43 // Actions are executed in parallel where the dependency graph allows.
44 // Overall parallelism is bounded by a semaphore, sized according to
45 // GOMAXPROCS. Each concurrently processed package takes up a
46 // token, as does each analyzer – but a package can always execute at
47 // least one analyzer, using the package's token.
48 //
49 // Depending on the overall shape of the graph, there may be GOMAXPROCS
50 // packages running a single analyzer each, a single package running
51 // GOMAXPROCS analyzers, or anything in between.
52 //
53 // Total memory consumption grows roughly linearly with the number of
54 // CPUs, while total execution time is inversely proportional to the
55 // number of CPUs. Overall, parallelism is affected by the shape of
56 // the dependency graph. A lot of inter-connected packages will see
57 // less parallelism than a lot of independent packages.
58 //
59 // # Caching
60 //
61 // The runner caches facts, directives and diagnostics in a
62 // content-addressable cache that is designed after Go's own cache.
63 // Additionally, it makes use of Go's export data.
64 //
65 // This cache not only speeds up repeat runs, it also reduces peak
66 // memory usage. When we've analyzed a package, we cache the results
67 // and drop them from memory. When a dependent needs any of this
68 // information, or when analysis is complete and we wish to render the
69 // results, the data gets loaded from disk again.
70 //
71 // Data only exists in memory when it is immediately needed, not
72 // retained for possible future uses. This trades increased CPU usage
73 // for reduced memory usage. A single dependency may be loaded many
74 // times over, but it greatly reduces peak memory usage, as an
75 // arbitrary amount of time may pass between analyzing a dependency
76 // and its dependent, during which other packages will be processed.
77 package runner
78
79 // OPT(dh): we could reduce disk storage usage of cached data by
80 // compressing it, either directly at the cache layer, or by feeding
81 // compressed data to the cache. Of course doing so may negatively
82 // affect CPU usage, and there are lower hanging fruit, such as
83 // needing to cache less data in the first place.
84
85 // OPT(dh): right now, each package is analyzed completely
86 // independently. Each package loads all of its dependencies from
87 // export data and cached facts. If we have two packages A and B,
88 // which both depend on C, and which both get analyzed in parallel,
89 // then C will be loaded twice. This wastes CPU time and memory. It
90 // would be nice if we could reuse a single C for the analysis of both
91 // A and B.
92 //
93 // We can't reuse the actual types.Package or facts, because each
94 // package gets its own token.FileSet. Sharing a global FileSet has
95 // several drawbacks, including increased memory usage and running the
96 // risk of running out of FileSet address space.
97 //
98 // We could however avoid loading the same raw export data from disk
99 // twice, as well as deserializing gob data twice. One possible
100 // solution would be a duplicate-suppressing in-memory cache that
101 // caches data for a limited amount of time. When the same package
102 // needs to be loaded twice in close succession, we can reuse work,
103 // without holding unnecessary data in memory for an extended period
104 // of time.
105 //
106 // We would likely need to do extensive benchmarking to figure out how
107 // long to keep data around to find a sweet spot where we reduce CPU
108 // load without increasing memory usage.
109 //
110 // We can probably populate the cache after we've analyzed a package,
111 // on the assumption that it will have to be loaded again in the near
112 // future.
113
114 import (
115 "bytes"
116 "encoding/gob"
117 "fmt"
118 "go/token"
119 "go/types"
120 "io"
121 "maps"
122 "os"
123 "reflect"
124 "runtime"
125 "sort"
126 "strings"
127 "sync/atomic"
128 "time"
129
130 "honnef.co/go/tools/analysis/lint"
131 "honnef.co/go/tools/analysis/report"
132 "honnef.co/go/tools/config"
133 "honnef.co/go/tools/go/loader"
134 tsync "honnef.co/go/tools/internal/sync"
135 "honnef.co/go/tools/lintcmd/cache"
136 "honnef.co/go/tools/unused"
137
138 "golang.org/x/tools/go/analysis"
139 "golang.org/x/tools/go/packages"
140 "golang.org/x/tools/go/types/objectpath"
141 )
142
143 const sanityCheck = false
144
145 // Diagnostic is like go/analysis.Diagnostic, but with all token.Pos resolved to token.Position.
146 type Diagnostic struct {
147 Position token.Position
148 End token.Position
149 Category string
150 Message string
151
152 SuggestedFixes []SuggestedFix
153 Related []RelatedInformation
154 }
155
156 // RelatedInformation provides additional context for a diagnostic.
157 type RelatedInformation struct {
158 Position token.Position
159 End token.Position
160 Message string
161 }
162
163 type SuggestedFix struct {
164 Message string
165 TextEdits []TextEdit
166 }
167
168 type TextEdit struct {
169 Position token.Position
170 End token.Position
171 NewText []byte
172 }
173
174 // A Result describes the result of analyzing a single package.
175 //
176 // It holds references to cached diagnostics and directives. They can
177 // be loaded on demand with the Load method.
178 type Result struct {
179 Package *loader.PackageSpec
180 Config config.Config
181 Initial bool
182 Skipped bool
183
184 Failed bool
185 Errors []error
186 // Action results, path to file
187 results string
188 // Results relevant to testing, only set when test mode is enabled, path to file
189 testData string
190 }
191
192 type SerializedDirective struct {
193 Command string
194 Arguments []string
195 // The position of the comment
196 DirectivePosition token.Position
197 // The position of the node that the comment is attached to
198 NodePosition token.Position
199 }
200
201 func serializeDirective(dir lint.Directive, fset *token.FileSet) SerializedDirective {
202 return SerializedDirective{
203 Command: dir.Command,
204 Arguments: dir.Arguments,
205 DirectivePosition: report.DisplayPosition(fset, dir.Directive.Pos()),
206 NodePosition: report.DisplayPosition(fset, dir.Node.Pos()),
207 }
208 }
209
210 type ResultData struct {
211 Directives []SerializedDirective
212 Diagnostics []Diagnostic
213 Unused unused.Result
214 }
215
216 func (r Result) Load() (ResultData, error) {
217 if r.Failed {
218 panic("Load called on failed Result")
219 }
220 if r.results == "" {
221 // this package was only a dependency
222 return ResultData{}, nil
223 }
224 f, err := os.Open(r.results)
225 if err != nil {
226 return ResultData{}, fmt.Errorf("failed loading result: %w", err)
227 }
228 defer f.Close()
229 var out ResultData
230 err = gob.NewDecoder(f).Decode(&out)
231 return out, err
232 }
233
234 // TestData contains extra information about analysis runs that is only available in test mode.
235 type TestData struct {
236 // Facts contains facts produced by analyzers for a package.
237 // Unlike vetx, this list only contains facts specific to this package,
238 // not all facts for the transitive closure of dependencies.
239 Facts []TestFact
240 // List of files that were part of the package.
241 Files []string
242 }
243
244 // LoadTest returns data relevant to testing.
245 // It should only be called if Runner.TestMode was set to true.
246 func (r Result) LoadTest() (TestData, error) {
247 if r.Failed {
248 panic("Load called on failed Result")
249 }
250 if r.results == "" {
251 // this package was only a dependency
252 return TestData{}, nil
253 }
254 f, err := os.Open(r.testData)
255 if err != nil {
256 return TestData{}, fmt.Errorf("failed loading test data: %w", err)
257 }
258 defer f.Close()
259 var out TestData
260 err = gob.NewDecoder(f).Decode(&out)
261 return out, err
262 }
263
264 type action interface {
265 Deps() []action
266 Triggers() []action
267 DecrementPending() bool
268 MarkFailed()
269 IsFailed() bool
270 AddError(error)
271 }
272
273 type baseAction struct {
274 // Action description
275
276 deps []action
277 triggers []action
278 pending uint32
279
280 // Action results
281
282 // failed is set to true if the action couldn't be processed. This
283 // may either be due to an error specific to this action, in
284 // which case the errors field will be populated, or due to a
285 // dependency being marked as failed, in which case errors will be
286 // empty.
287 failed bool
288 errors []error
289 }
290
291 func (act *baseAction) Deps() []action { return act.deps }
292 func (act *baseAction) Triggers() []action { return act.triggers }
293 func (act *baseAction) DecrementPending() bool {
294 return atomic.AddUint32(&act.pending, ^uint32(0)) == 0
295 }
296 func (act *baseAction) MarkFailed() { act.failed = true }
297 func (act *baseAction) IsFailed() bool { return act.failed }
298 func (act *baseAction) AddError(err error) { act.errors = append(act.errors, err) }
299
300 // packageAction describes the act of loading a package, fully
301 // analyzing it, and storing the results.
302 type packageAction struct {
303 baseAction
304
305 // Action description
306 Package *loader.PackageSpec
307 factsOnly bool
308 hash cache.ActionID
309
310 // Action results
311 cfg config.Config
312 vetx string
313 results string
314 testData string
315 skipped bool
316 }
317
318 func (act *packageAction) String() string {
319 return fmt.Sprintf("packageAction(%s)", act.Package)
320 }
321
322 type objectFact struct {
323 fact analysis.Fact
324 // TODO(dh): why do we store the objectpath when producing the
325 // fact? Is it just for the sanity checking, which compares the
326 // stored path with a path recomputed from objectFactKey.Obj?
327 path objectpath.Path
328 }
329
330 type objectFactKey struct {
331 Obj types.Object
332 Type reflect.Type
333 }
334
335 type packageFactKey struct {
336 Pkg *types.Package
337 Type reflect.Type
338 }
339
340 type gobFact struct {
341 PkgPath string
342 ObjPath string
343 Fact analysis.Fact
344 }
345
346 // TestFact is a serialization of facts that is specific to the test mode.
347 type TestFact struct {
348 ObjectName string
349 Position token.Position
350 FactString string
351 Analyzer string
352 }
353
354 // analyzerAction describes the act of analyzing a package with a
355 // single analyzer.
356 type analyzerAction struct {
357 baseAction
358
359 // Action description
360
361 Analyzer *analysis.Analyzer
362
363 // Action results
364
365 // We can store actual results here without worrying about memory
366 // consumption because analyzer actions get garbage collected once
367 // a package has been fully analyzed.
368 Result any
369 Diagnostics []Diagnostic
370 ObjectFacts map[objectFactKey]objectFact
371 PackageFacts map[packageFactKey]analysis.Fact
372 Pass *analysis.Pass
373 }
374
375 func (act *analyzerAction) String() string {
376 return fmt.Sprintf("analyzerAction(%s)", act.Analyzer)
377 }
378
379 // A Runner executes analyzers on packages.
380 type Runner struct {
381 Stats Stats
382 GoVersion string
383
384 // If set to true, Runner will populate results with data relevant to testing analyzers
385 TestMode bool
386
387 // Config that gets merged with per-package configs
388 cfg config.Config
389 cache *cache.Cache
390 semaphore tsync.Semaphore
391 }
392
393 type subrunner struct {
394 *Runner
395 analyzers []*analysis.Analyzer
396 factAnalyzers []*analysis.Analyzer
397 analyzerNames string
398 cache *cache.Cache
399 }
400
401 // New returns a new Runner.
402 func New(cfg config.Config, c *cache.Cache) (*Runner, error) {
403 return &Runner{
404 cfg: cfg,
405 cache: c,
406 semaphore: tsync.NewSemaphore(runtime.GOMAXPROCS(0)),
407 }, nil
408 }
409
410 func newSubrunner(r *Runner, analyzers []*analysis.Analyzer) *subrunner {
411 analyzerNames := make([]string, len(analyzers))
412 for i, a := range analyzers {
413 analyzerNames[i] = a.Name
414 }
415 sort.Strings(analyzerNames)
416
417 var factAnalyzers []*analysis.Analyzer
418 for _, a := range analyzers {
419 if len(a.FactTypes) > 0 {
420 factAnalyzers = append(factAnalyzers, a)
421 }
422 }
423 return &subrunner{
424 Runner: r,
425 analyzers: analyzers,
426 factAnalyzers: factAnalyzers,
427 analyzerNames: strings.Join(analyzerNames, ","),
428 cache: r.cache,
429 }
430 }
431
432 func newPackageActionRoot(pkg *loader.PackageSpec, cache map[*loader.PackageSpec]*packageAction) *packageAction {
433 a := newPackageAction(pkg, cache)
434 a.factsOnly = false
435 return a
436 }
437
438 func newPackageAction(pkg *loader.PackageSpec, cache map[*loader.PackageSpec]*packageAction) *packageAction {
439 if a, ok := cache[pkg]; ok {
440 return a
441 }
442
443 a := &packageAction{
444 Package: pkg,
445 factsOnly: true, // will be overwritten by any call to Action
446 }
447 cache[pkg] = a
448
449 if len(pkg.Errors) > 0 {
450 a.errors = make([]error, len(pkg.Errors))
451 for i, err := range pkg.Errors {
452 a.errors[i] = err
453 }
454 a.failed = true
455
456 // We don't need to process our imports if this package is
457 // already broken.
458 return a
459 }
460
461 a.deps = make([]action, 0, len(pkg.Imports))
462 for _, dep := range pkg.Imports {
463 depa := newPackageAction(dep, cache)
464 depa.triggers = append(depa.triggers, a)
465 a.deps = append(a.deps, depa)
466
467 if depa.failed {
468 a.failed = true
469 }
470 }
471 // sort dependencies because the list of dependencies is part of
472 // the cache key
473 sort.Slice(a.deps, func(i, j int) bool {
474 return a.deps[i].(*packageAction).Package.ID < a.deps[j].(*packageAction).Package.ID
475 })
476
477 a.pending = uint32(len(a.deps))
478
479 return a
480 }
481
482 func newAnalyzerAction(an *analysis.Analyzer, cache map[*analysis.Analyzer]*analyzerAction) *analyzerAction {
483 if a, ok := cache[an]; ok {
484 return a
485 }
486
487 a := &analyzerAction{
488 Analyzer: an,
489 ObjectFacts: map[objectFactKey]objectFact{},
490 PackageFacts: map[packageFactKey]analysis.Fact{},
491 }
492 cache[an] = a
493 for _, dep := range an.Requires {
494 depa := newAnalyzerAction(dep, cache)
495 depa.triggers = append(depa.triggers, a)
496 a.deps = append(a.deps, depa)
497 }
498 a.pending = uint32(len(a.deps))
499 return a
500 }
501
502 func getCachedFiles(cache *cache.Cache, ids []cache.ActionID, out []*string) error {
503 for i, id := range ids {
504 var err error
505 *out[i], _, err = cache.GetFile(id)
506 if err != nil {
507 return err
508 }
509 }
510 return nil
511 }
512
513 func (r *subrunner) do(act action) error {
514 a := act.(*packageAction)
515 defer func() {
516 r.Stats.finishPackage()
517 if !a.factsOnly {
518 r.Stats.finishInitialPackage()
519 }
520 }()
521
522 // compute hash of action
523 a.cfg = a.Package.Config.Merge(r.cfg)
524 h := r.cache.NewHash("staticcheck " + a.Package.PkgPath)
525
526 // Note that we do not filter the list of analyzers by the
527 // package's configuration. We don't allow configuration to
528 // accidentally break dependencies between analyzers, and it's
529 // easier to always run all checks and filter the output. This
530 // also makes cached data more reusable.
531
532 // OPT(dh): not all changes in configuration invalidate cached
533 // data. specifically, when a.factsOnly == true, we only care
534 // about checks that produce facts, and settings that affect those
535 // checks.
536
537 // Config used for constructing the hash; this config doesn't have
538 // Checks populated, because we always run all checks.
539 //
540 // This even works for users who add custom checks, because we include the binary's hash.
541 hashCfg := a.cfg
542 hashCfg.Checks = nil
543 // note that we don't hash staticcheck's version; it is set as the
544 // salt by a package main.
545 fmt.Fprintf(h, "cfg %#v\n", hashCfg)
546 fmt.Fprintf(h, "pkg %x\n", a.Package.Hash)
547 fmt.Fprintf(h, "analyzers %s\n", r.analyzerNames)
548 fmt.Fprintf(h, "go %s\n", r.GoVersion)
549 fmt.Fprintf(h, "env godebug %q\n", os.Getenv("GODEBUG"))
550
551 // OPT(dh): do we actually need to hash vetx? can we not assume
552 // that for identical inputs, staticcheck will produce identical
553 // vetx?
554 for _, dep := range a.deps {
555 dep := dep.(*packageAction)
556 vetxHash, err := cache.FileHash(dep.vetx)
557 if err != nil {
558 return fmt.Errorf("failed computing hash: %w", err)
559 }
560 fmt.Fprintf(h, "vetout %q %x\n", dep.Package.PkgPath, vetxHash)
561 }
562 a.hash = cache.ActionID(h.Sum())
563
564 // try to fetch hashed data
565 ids := make([]cache.ActionID, 0, 2)
566 ids = append(ids, cache.Subkey(a.hash, "vetx"))
567 if !a.factsOnly {
568 ids = append(ids, cache.Subkey(a.hash, "results"))
569 if r.TestMode {
570 ids = append(ids, cache.Subkey(a.hash, "testdata"))
571 }
572 }
573 if err := getCachedFiles(r.cache, ids, []*string{&a.vetx, &a.results, &a.testData}); err != nil {
574 result, err := r.doUncached(a)
575 if err != nil {
576 return err
577 }
578 if a.failed {
579 return nil
580 }
581
582 a.skipped = result.skipped
583
584 // OPT(dh) instead of collecting all object facts and encoding
585 // them after analysis finishes, we could encode them as we
586 // go. however, that would require some locking.
587 //
588 // OPT(dh): We could sort gobFacts for more consistent output,
589 // but it doesn't matter. The hash of a package includes all
590 // of its files, so whether the vetx hash changes or not, a
591 // change to a package requires re-analyzing all dependents,
592 // even if the vetx data stayed the same. See also the note at
593 // the top of loader/hash.go.
594
595 tf := &bytes.Buffer{}
596 enc := gob.NewEncoder(tf)
597 for _, gf := range result.facts {
598 if err := enc.Encode(gf); err != nil {
599 return fmt.Errorf("failed gob encoding data: %w", err)
600 }
601 }
602
603 a.vetx, err = r.writeCacheReader(a, "vetx", bytes.NewReader(tf.Bytes()))
604 if err != nil {
605 return err
606 }
607
608 if a.factsOnly {
609 return nil
610 }
611
612 var out ResultData
613 out.Directives = make([]SerializedDirective, len(result.dirs))
614 for i, dir := range result.dirs {
615 out.Directives[i] = serializeDirective(dir, result.lpkg.Fset)
616 }
617
618 out.Diagnostics = result.diags
619 out.Unused = result.unused
620 a.results, err = r.writeCacheGob(a, "results", out)
621 if err != nil {
622 return err
623 }
624
625 if r.TestMode {
626 out := TestData{
627 Facts: result.testFacts,
628 Files: result.lpkg.GoFiles,
629 }
630 a.testData, err = r.writeCacheGob(a, "testdata", out)
631 if err != nil {
632 return err
633 }
634 }
635 }
636 return nil
637 }
638
639 // ActiveWorkers returns the number of currently running workers.
640 func (r *Runner) ActiveWorkers() int {
641 return r.semaphore.Len()
642 }
643
644 // TotalWorkers returns the maximum number of possible workers.
645 func (r *Runner) TotalWorkers() int {
646 return r.semaphore.Cap()
647 }
648
649 func (r *Runner) writeCacheReader(a *packageAction, kind string, rs io.ReadSeeker) (string, error) {
650 h := cache.Subkey(a.hash, kind)
651 out, _, err := r.cache.Put(h, rs)
652 if err != nil {
653 return "", fmt.Errorf("failed caching data: %w", err)
654 }
655 return r.cache.OutputFile(out), nil
656 }
657
658 func (r *Runner) writeCacheGob(a *packageAction, kind string, data any) (string, error) {
659 f, err := os.CreateTemp("", "staticcheck")
660 if err != nil {
661 return "", err
662 }
663 defer f.Close()
664 os.Remove(f.Name())
665 if err := gob.NewEncoder(f).Encode(data); err != nil {
666 return "", fmt.Errorf("failed gob encoding data: %w", err)
667 }
668 if _, err := f.Seek(0, io.SeekStart); err != nil {
669 return "", err
670 }
671 return r.writeCacheReader(a, kind, f)
672 }
673
674 type packageActionResult struct {
675 facts []gobFact
676 diags []Diagnostic
677 unused unused.Result
678 dirs []lint.Directive
679 lpkg *loader.Package
680 skipped bool
681
682 // Only set when using test mode
683 testFacts []TestFact
684 }
685
686 func (r *subrunner) doUncached(a *packageAction) (packageActionResult, error) {
687 // OPT(dh): for a -> b; c -> b; if both a and b are being
688 // processed concurrently, we shouldn't load b's export data
689 // twice.
690
691 pkg, _, err := loader.Load(a.Package, &loader.Options{GoVersion: r.GoVersion})
692 if err != nil {
693 return packageActionResult{}, err
694 }
695
696 if len(pkg.Errors) > 0 {
697 // this handles errors that occurred during type-checking the
698 // package in loader.Load
699 for _, err := range pkg.Errors {
700 a.errors = append(a.errors, err)
701 }
702 a.failed = true
703 return packageActionResult{}, nil
704 }
705
706 if len(pkg.Syntax) == 0 && pkg.PkgPath != "unsafe" {
707 return packageActionResult{lpkg: pkg, skipped: true}, nil
708 }
709
710 // OPT(dh): instead of parsing directives twice (twice because
711 // U1000 depends on the facts.Directives analyzer), reuse the
712 // existing result
713 var dirs []lint.Directive
714 if !a.factsOnly {
715 dirs = lint.ParseDirectives(pkg.Syntax, pkg.Fset)
716 }
717 res, err := r.runAnalyzers(a, pkg)
718
719 return packageActionResult{
720 facts: res.facts,
721 testFacts: res.testFacts,
722 diags: res.diagnostics,
723 unused: res.unused,
724 dirs: dirs,
725 lpkg: pkg,
726 }, err
727 }
728
729 func pkgPaths(root *types.Package) map[string]*types.Package {
730 out := map[string]*types.Package{}
731 var dfs func(*types.Package)
732 dfs = func(pkg *types.Package) {
733 if _, ok := out[pkg.Path()]; ok {
734 return
735 }
736 out[pkg.Path()] = pkg
737 for _, imp := range pkg.Imports() {
738 dfs(imp)
739 }
740 }
741 dfs(root)
742 return out
743 }
744
745 func (r *Runner) loadFacts(root *types.Package, dep *packageAction, objFacts map[objectFactKey]objectFact, pkgFacts map[packageFactKey]analysis.Fact) error {
746 // Load facts of all imported packages
747 vetx, err := os.Open(dep.vetx)
748 if err != nil {
749 return fmt.Errorf("failed loading cached facts: %w", err)
750 }
751 defer vetx.Close()
752
753 pathToPkg := pkgPaths(root)
754 dec := gob.NewDecoder(vetx)
755 for {
756 var gf gobFact
757 err := dec.Decode(&gf)
758 if err != nil {
759 if err == io.EOF {
760 break
761 }
762 return fmt.Errorf("failed loading cached facts: %w", err)
763 }
764
765 pkg, ok := pathToPkg[gf.PkgPath]
766 if !ok {
767 continue
768 }
769 if gf.ObjPath == "" {
770 pkgFacts[packageFactKey{
771 Pkg: pkg,
772 Type: reflect.TypeOf(gf.Fact),
773 }] = gf.Fact
774 } else {
775 obj, err := objectpath.Object(pkg, objectpath.Path(gf.ObjPath))
776 if err != nil {
777 continue
778 }
779 objFacts[objectFactKey{
780 Obj: obj,
781 Type: reflect.TypeOf(gf.Fact),
782 }] = objectFact{gf.Fact, objectpath.Path(gf.ObjPath)}
783 }
784 }
785 return nil
786 }
787
788 func genericHandle(a action, root action, queue chan action, sem *tsync.Semaphore, exec func(a action) error) {
789 if a == root {
790 close(queue)
791 if sem != nil {
792 sem.Release()
793 }
794 return
795 }
796 if !a.IsFailed() {
797 // the action may have already been marked as failed during
798 // construction of the action graph, for example because of
799 // unresolved imports.
800
801 for _, dep := range a.Deps() {
802 if dep.IsFailed() {
803 // One of our dependencies failed, so mark this package as
804 // failed and bail. We don't need to record an error for
805 // this package, the relevant error will have been
806 // reported by the first package in the chain that failed.
807 a.MarkFailed()
808 break
809 }
810 }
811 }
812
813 if !a.IsFailed() {
814 if err := exec(a); err != nil {
815 a.MarkFailed()
816 a.AddError(err)
817 }
818 }
819 if sem != nil {
820 sem.Release()
821 }
822
823 for _, t := range a.Triggers() {
824 if t.DecrementPending() {
825 queue <- t
826 }
827 }
828 }
829
830 type analyzerRunner struct {
831 pkg *loader.Package
832 // object facts of our dependencies; may contain facts of
833 // analyzers other than the current one
834 depObjFacts map[objectFactKey]objectFact
835 // package facts of our dependencies; may contain facts of
836 // analyzers other than the current one
837 depPkgFacts map[packageFactKey]analysis.Fact
838 factsOnly bool
839
840 stats *Stats
841 }
842
843 func (ar *analyzerRunner) do(act action) error {
844 a := act.(*analyzerAction)
845 results := map[*analysis.Analyzer]any{}
846 // TODO(dh): does this have to be recursive?
847 for _, dep := range a.deps {
848 dep := dep.(*analyzerAction)
849 results[dep.Analyzer] = dep.Result
850 }
851 // OPT(dh): cache factTypes, it is the same for all packages for a given analyzer
852 //
853 // OPT(dh): do we need the factTypes map? most analyzers have 0-1
854 // fact types. iterating over the slice is probably faster than
855 // indexing a map.
856 factTypes := map[reflect.Type]struct{}{}
857 for _, typ := range a.Analyzer.FactTypes {
858 factTypes[reflect.TypeOf(typ)] = struct{}{}
859 }
860 filterFactType := func(typ reflect.Type) bool {
861 _, ok := factTypes[typ]
862 return ok
863 }
864 a.Pass = &analysis.Pass{
865 Analyzer: a.Analyzer,
866 Fset: ar.pkg.Fset,
867 Files: ar.pkg.Syntax,
868 OtherFiles: ar.pkg.OtherFiles,
869 Pkg: ar.pkg.Types,
870 TypesInfo: ar.pkg.TypesInfo,
871 TypesSizes: ar.pkg.TypesSizes,
872 Report: func(diag analysis.Diagnostic) {
873 if !ar.factsOnly {
874 if diag.Category == "" {
875 diag.Category = a.Analyzer.Name
876 }
877 d := Diagnostic{
878 Position: report.DisplayPosition(ar.pkg.Fset, diag.Pos),
879 End: report.DisplayPosition(ar.pkg.Fset, diag.End),
880 Category: diag.Category,
881 Message: diag.Message,
882 }
883 for _, sugg := range diag.SuggestedFixes {
884 s := SuggestedFix{
885 Message: sugg.Message,
886 }
887 for _, edit := range sugg.TextEdits {
888 s.TextEdits = append(s.TextEdits, TextEdit{
889 Position: report.DisplayPosition(ar.pkg.Fset, edit.Pos),
890 End: report.DisplayPosition(ar.pkg.Fset, edit.End),
891 NewText: edit.NewText,
892 })
893 }
894 d.SuggestedFixes = append(d.SuggestedFixes, s)
895 }
896 for _, rel := range diag.Related {
897 d.Related = append(d.Related, RelatedInformation{
898 Position: report.DisplayPosition(ar.pkg.Fset, rel.Pos),
899 End: report.DisplayPosition(ar.pkg.Fset, rel.End),
900 Message: rel.Message,
901 })
902 }
903 a.Diagnostics = append(a.Diagnostics, d)
904 }
905 },
906 ResultOf: results,
907 ImportObjectFact: func(obj types.Object, fact analysis.Fact) bool {
908 key := objectFactKey{
909 Obj: obj,
910 Type: reflect.TypeOf(fact),
911 }
912 if f, ok := ar.depObjFacts[key]; ok {
913 reflect.ValueOf(fact).Elem().Set(reflect.ValueOf(f.fact).Elem())
914 return true
915 } else if f, ok := a.ObjectFacts[key]; ok {
916 reflect.ValueOf(fact).Elem().Set(reflect.ValueOf(f.fact).Elem())
917 return true
918 }
919 return false
920 },
921 ImportPackageFact: func(pkg *types.Package, fact analysis.Fact) bool {
922 key := packageFactKey{
923 Pkg: pkg,
924 Type: reflect.TypeOf(fact),
925 }
926 if f, ok := ar.depPkgFacts[key]; ok {
927 reflect.ValueOf(fact).Elem().Set(reflect.ValueOf(f).Elem())
928 return true
929 } else if f, ok := a.PackageFacts[key]; ok {
930 reflect.ValueOf(fact).Elem().Set(reflect.ValueOf(f).Elem())
931 return true
932 }
933 return false
934 },
935 ExportObjectFact: func(obj types.Object, fact analysis.Fact) {
936 key := objectFactKey{
937 Obj: obj,
938 Type: reflect.TypeOf(fact),
939 }
940 path, _ := objectpath.For(obj)
941 a.ObjectFacts[key] = objectFact{fact, path}
942 },
943 ExportPackageFact: func(fact analysis.Fact) {
944 key := packageFactKey{
945 Pkg: ar.pkg.Types,
946 Type: reflect.TypeOf(fact),
947 }
948 a.PackageFacts[key] = fact
949 },
950 AllPackageFacts: func() []analysis.PackageFact {
951 out := make([]analysis.PackageFact, 0, len(ar.depPkgFacts)+len(a.PackageFacts))
952 for key, fact := range ar.depPkgFacts {
953 out = append(out, analysis.PackageFact{
954 Package: key.Pkg,
955 Fact: fact,
956 })
957 }
958 for key, fact := range a.PackageFacts {
959 out = append(out, analysis.PackageFact{
960 Package: key.Pkg,
961 Fact: fact,
962 })
963 }
964 return out
965 },
966 AllObjectFacts: func() []analysis.ObjectFact {
967 out := make([]analysis.ObjectFact, 0, len(ar.depObjFacts)+len(a.ObjectFacts))
968 for key, fact := range ar.depObjFacts {
969 if filterFactType(key.Type) {
970 out = append(out, analysis.ObjectFact{
971 Object: key.Obj,
972 Fact: fact.fact,
973 })
974 }
975 }
976 for key, fact := range a.ObjectFacts {
977 if filterFactType(key.Type) {
978 out = append(out, analysis.ObjectFact{
979 Object: key.Obj,
980 Fact: fact.fact,
981 })
982 }
983 }
984 return out
985 },
986 }
987
988 t := time.Now()
989 res, err := a.Analyzer.Run(a.Pass)
990 ar.stats.measureAnalyzer(a.Analyzer, ar.pkg.PackageSpec, time.Since(t))
991 if err != nil {
992 return err
993 }
994 a.Result = res
995 return nil
996 }
997
998 type analysisResult struct {
999 facts []gobFact
1000 diagnostics []Diagnostic
1001 unused unused.Result
1002
1003 // Only set when using test mode
1004 testFacts []TestFact
1005 }
1006
1007 func (r *subrunner) runAnalyzers(pkgAct *packageAction, pkg *loader.Package) (analysisResult, error) {
1008 depObjFacts := map[objectFactKey]objectFact{}
1009 depPkgFacts := map[packageFactKey]analysis.Fact{}
1010
1011 for _, dep := range pkgAct.deps {
1012 if err := r.loadFacts(pkg.Types, dep.(*packageAction), depObjFacts, depPkgFacts); err != nil {
1013 return analysisResult{}, err
1014 }
1015 }
1016
1017 root := &analyzerAction{}
1018 var analyzers []*analysis.Analyzer
1019 if pkgAct.factsOnly {
1020 // When analyzing non-initial packages, we only care about
1021 // analyzers that produce facts.
1022 analyzers = r.factAnalyzers
1023 } else {
1024 analyzers = r.analyzers
1025 }
1026
1027 all := map[*analysis.Analyzer]*analyzerAction{}
1028 for _, a := range analyzers {
1029 a := newAnalyzerAction(a, all)
1030 root.deps = append(root.deps, a)
1031 a.triggers = append(a.triggers, root)
1032 }
1033 root.pending = uint32(len(root.deps))
1034
1035 ar := &analyzerRunner{
1036 pkg: pkg,
1037 factsOnly: pkgAct.factsOnly,
1038 depObjFacts: depObjFacts,
1039 depPkgFacts: depPkgFacts,
1040 stats: &r.Stats,
1041 }
1042 queue := make(chan action, len(all))
1043 for _, a := range all {
1044 if len(a.Deps()) == 0 {
1045 queue <- a
1046 }
1047 }
1048
1049 // Don't hang if there are no analyzers to run; for example
1050 // because we are analyzing a dependency but have no analyzers
1051 // that produce facts.
1052 if len(all) == 0 {
1053 close(queue)
1054 }
1055 for item := range queue {
1056 b := r.semaphore.AcquireMaybe()
1057 if b {
1058 go genericHandle(item, root, queue, &r.semaphore, ar.do)
1059 } else {
1060 // the semaphore is exhausted; run the analysis under the
1061 // token we've acquired for analyzing the package.
1062 genericHandle(item, root, queue, nil, ar.do)
1063 }
1064 }
1065
1066 var unusedResult unused.Result
1067 for _, a := range all {
1068 if a != root && a.Analyzer.Name == "U1000" && !a.failed {
1069 // TODO(dh): figure out a clean abstraction, instead of
1070 // special-casing U1000.
1071 unusedResult = a.Result.(unused.Result)
1072 }
1073
1074 maps.Copy(depObjFacts, a.ObjectFacts)
1075 maps.Copy(depPkgFacts, a.PackageFacts)
1076 }
1077
1078 // OPT(dh): cull objects not reachable via the exported closure
1079 var testFacts []TestFact
1080 gobFacts := make([]gobFact, 0, len(depObjFacts)+len(depPkgFacts))
1081 for key, fact := range depObjFacts {
1082 if fact.path == "" {
1083 continue
1084 }
1085 if sanityCheck {
1086 p, _ := objectpath.For(key.Obj)
1087 if p != fact.path {
1088 panic(fmt.Sprintf("got different object paths for %v. old: %q new: %q", key.Obj, fact.path, p))
1089 }
1090 }
1091 gf := gobFact{
1092 PkgPath: key.Obj.Pkg().Path(),
1093 ObjPath: string(fact.path),
1094 Fact: fact.fact,
1095 }
1096 gobFacts = append(gobFacts, gf)
1097 }
1098
1099 for key, fact := range depPkgFacts {
1100 gf := gobFact{
1101 PkgPath: key.Pkg.Path(),
1102 Fact: fact,
1103 }
1104 gobFacts = append(gobFacts, gf)
1105 }
1106
1107 if r.TestMode {
1108 for _, a := range all {
1109 for key, fact := range a.ObjectFacts {
1110 tgf := TestFact{
1111 ObjectName: key.Obj.Name(),
1112 Position: pkg.Fset.Position(key.Obj.Pos()),
1113 FactString: fmt.Sprint(fact.fact),
1114 Analyzer: a.Analyzer.Name,
1115 }
1116 testFacts = append(testFacts, tgf)
1117 }
1118
1119 for _, fact := range a.PackageFacts {
1120 tgf := TestFact{
1121 ObjectName: "",
1122 Position: pkg.Fset.Position(pkg.Syntax[0].Pos()),
1123 FactString: fmt.Sprint(fact),
1124 Analyzer: a.Analyzer.Name,
1125 }
1126 testFacts = append(testFacts, tgf)
1127 }
1128 }
1129 }
1130
1131 var diags []Diagnostic
1132 for _, a := range root.deps {
1133 a := a.(*analyzerAction)
1134 diags = append(diags, a.Diagnostics...)
1135 }
1136 return analysisResult{
1137 facts: gobFacts,
1138 testFacts: testFacts,
1139 diagnostics: diags,
1140 unused: unusedResult,
1141 }, nil
1142 }
1143
1144 func registerGobTypes(analyzers []*analysis.Analyzer) {
1145 for _, a := range analyzers {
1146 for _, typ := range a.FactTypes {
1147 // FIXME(dh): use RegisterName so we can work around collisions
1148 // in names. For pointer-types, gob incorrectly qualifies
1149 // type names with the package name, not the import path.
1150 gob.Register(typ)
1151 }
1152 }
1153 }
1154
1155 func allAnalyzers(analyzers []*analysis.Analyzer) []*analysis.Analyzer {
1156 seen := map[*analysis.Analyzer]struct{}{}
1157 out := make([]*analysis.Analyzer, 0, len(analyzers))
1158 var dfs func(*analysis.Analyzer)
1159 dfs = func(a *analysis.Analyzer) {
1160 if _, ok := seen[a]; ok {
1161 return
1162 }
1163 seen[a] = struct{}{}
1164 out = append(out, a)
1165 for _, dep := range a.Requires {
1166 dfs(dep)
1167 }
1168 }
1169 for _, a := range analyzers {
1170 dfs(a)
1171 }
1172 return out
1173 }
1174
1175 // Run loads the packages specified by patterns, runs analyzers on
1176 // them and returns the results. Each result corresponds to a single
1177 // package. Results will be returned for all packages, including
1178 // dependencies. Errors specific to packages will be reported in the
1179 // respective results.
1180 //
1181 // If cfg is nil, a default config will be used. Otherwise, cfg will
1182 // be used, with the exception of the Mode field.
1183 func (r *Runner) Run(cfg *packages.Config, analyzers []*analysis.Analyzer, patterns []string) ([]Result, error) {
1184 analyzers = allAnalyzers(analyzers)
1185 registerGobTypes(analyzers)
1186
1187 r.Stats.setState(StateLoadPackageGraph)
1188 lpkgs, err := loader.Graph(r.cache, cfg, patterns...)
1189 if err != nil {
1190 return nil, err
1191 }
1192 r.Stats.setInitialPackages(len(lpkgs))
1193
1194 if len(lpkgs) == 0 {
1195 return nil, nil
1196 }
1197
1198 r.Stats.setState(StateBuildActionGraph)
1199 all := map[*loader.PackageSpec]*packageAction{}
1200 root := &packageAction{}
1201 for _, lpkg := range lpkgs {
1202 a := newPackageActionRoot(lpkg, all)
1203 root.deps = append(root.deps, a)
1204 a.triggers = append(a.triggers, root)
1205 }
1206 root.pending = uint32(len(root.deps))
1207
1208 queue := make(chan action)
1209 r.Stats.setTotalPackages(len(all) - 1)
1210
1211 r.Stats.setState(StateProcessing)
1212 go func() {
1213 for _, a := range all {
1214 if len(a.Deps()) == 0 {
1215 queue <- a
1216 }
1217 }
1218 }()
1219
1220 sr := newSubrunner(r, analyzers)
1221 for item := range queue {
1222 r.semaphore.Acquire()
1223 go genericHandle(item, root, queue, &r.semaphore, func(act action) error {
1224 return sr.do(act)
1225 })
1226 }
1227
1228 r.Stats.setState(StateFinalizing)
1229 out := make([]Result, 0, len(all))
1230 for _, item := range all {
1231 if item.Package == nil {
1232 continue
1233 }
1234 out = append(out, Result{
1235 Package: item.Package,
1236 Config: item.cfg,
1237 Initial: !item.factsOnly,
1238 Skipped: item.skipped,
1239 Failed: item.failed,
1240 Errors: item.errors,
1241 results: item.results,
1242 testData: item.testData,
1243 })
1244 }
1245 return out, nil
1246 }